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Abstract 


Multiple  linear  regression  techniques  were  employed  to  examine  source 
selection  in  an  Air  Force  procurement  division.  The  experimental  data 
was  derived  from  an  exercise  which  examined  a  part  of  the  source  selection 
process.  Each  exercise  contained  identical  Request  for  Proposals.  Sub¬ 
jects  were  asked  to  rate  proposals  with  and  without  a  policy  (prioritizing 
evaluation  criteria).  The  results  of  the  exercise  were  grouped  three 
ways:  (1)  by  organization,  (?)  by  supervisory  level,  and  (3)  by  experi¬ 
ence  with  source  selection.  All  possible  pairings  within  the  above  three 
categories  were  then  compared  to  identify  any  significant  differences 
between  sub-category  members.  All  groupings  were  investigated  to  deter¬ 
mine  whether  or  not  each  evaluation  criteria  contributed  significantly 
to  the  rating  a  proposal  received.  Assimilation  of  the  policy  (priori¬ 
tizing  evaluation  criteria)  was  examined.  Last,  the  linear  model  was 
compared  to  a  non-linear  model  to  determine  whether  or  not  the  linear 
model  was  adequate.  — {■■  ; 

The  linear  model  was  proven  to  be  acceptable  in  modeling  this  source 
selection  decision.  Significant  differences  in  how  each  oiganization 
rated  proposals  were  discovered.  Supervisory  level  did  not  yield  any 
significant  differences  in  rating  proposals.  There  were  differences  in 
how  those  who  had  participated  in  source  selection  and  those  who  had 
not  rated  proposals.  All  of  the  evaluation  criteria  did  not  contribute 
significantly  to  proposal  rating  when  subjects  were  given  a  prioritized 
listing  of  evaluation  criteria  or  policy.  There  was  no  difficulty  in 
assimilating  a  policy  and  no  significant  difference  in  degree  of  policy 


assimilation.  The*»  results  suggest  that  source  selection  decisions  are 
not  similar  across  organizations  within  the  Air  Force  division.  Further¬ 
more,  subjects  did  not  utilize  all  the  information  available  to  them  in 

\ 

making  decisions.  People  often  chose  to  utilize  only  a  part  of  the 
available  information  in  arriving  at  a  decision. 


A  CRITICAL  APPRAISAL  OF  SOURCE  SELECTION  PROCEDURES 


CHAPTER  I 
INTRODUCTION 

Source  selection  is  an  activity  performed  by  many  systems  program 
managers.  While  it  represents  only  one  of  the  tasks  associated  with 
program  management,  it  is  perhaps  one  of  the  most  interesting  aspects  of 
the  job.  When  accomplishing  a  procurement  limited  to  a  less  than  "major" 
dollar  amount,  the  source  selection  process  affords  program  managers  the 
opportunity  to  exercise  judgment  and  exert  influence.  This  opportunity 
is  not  available  in  some  of  the  other  tasks  associated  with  program 
management.  The  authority  is  vested  in  higher  levels  of  management  as 
the  dollar  amount  of  a  program  increases. 

This  thesis  will  examine  actions  of  program  managers  and  supervi¬ 
sors  in  the  Deputy  for  Aeronautical  Equipment  (AE),  Aeronautical  Systems 
Division  (ASD),  Wright-Patterson  Air  Force  Base,  Ohio.  Major  concentra¬ 
tion  will  be  on  examining  how  source  selection  decisions  are  made  when 
authority  has  been  delegated  as  far  downward  as  possible.  The  area  of 
interest  is  source  selection  decisions  under  conditions  of  maximum 
freedom.  Maximum  freedom  is  attained  where  less  than  "majoi  "  programs 
are  to  be  procured.  In  Table  I,  the  thresholds,  which  determine  the 
delegation  of  source  selection  authority  (SSA)  for  AE,  are  presented. 

Source  Select ion 

Source  selection  specifically  includes  "...  soliciting  and  eval¬ 
uating  offerors'  proposals  and  .  .  .  selecting  sources  for  development 
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TABLE  I 


Delegation  of  Source  Selection  Authority’ 


Potential  Program  Cos t 

$0  -  $500,000  RDT&E  and/or 
$0  -  $5,000,000  Production 

$500,001  -  $1,000,003  RDT&E 
and/or  $5,000,001  - 
$7,500,000  Production 

$1,000,001  -  $12,500,000 
RDT&E  and/or  $7,500,001  - 
$25,000,000  Production 

$12,500,001  -  $25,000,000 
RDT&E  and/or  $25,000,001  - 
$50,000,000  Production 


*AE  Operating  Instruction  70-2 


So urce  Se  1  ecti on  A u t! f lority 

Procuring  Contracting  Officer  (PC0) 
without  power  of  redelegation 

Chief  of  the  Procurement  Division 
of  the  3  letter  organization 
without  power  of  redt legation 

Director/Deputy  Director/Assistant 
Director  of  the  3  letter  organi¬ 
zations  without  pov.er  of  redelegation 

Deputy/Assistant  Deputy 
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and  production  of  major  defense  systems,  subsystems,  and  components  as 
weli  as  oilier  major  programs  ot  projects  competitively  procured  by  the 
Department  of  the  Air  Force"  (AIT<  70-lb,  1976:1).  Source  Selection 
Policy  and  Procedures,  APR  70-15,  establishes  policy,  assigns  authority 
and  responsibilities,  and  prescribes  implementing  procedures  for  source 
selection  (APR  70-15,  1976:1). 

AFR  70-15  establishes  procedures  for  "major"  programs.  A  major 
program  requires  $75  million  or  more  Research,  Development,  Test,  and 
Evaluation  (RDT&E)  funds  or  is  projected  to  require  $300  million  or  more 
production  funds  (including  support)  (AFR  70-15,  1976:1-1).  There  are 
other  dollar  thresholds  which,  when  exceeded,  require  the  application 
of  AFR  70-15.  This  thesis  will  consider  source  selection  requiring  funds 
which  do  not  exceed  these  APR  70-15  thresholds. 

An  explanation  of  terms  is  now  presented  in  order  to  enhance  an 
overall  understanding  of  the  source  selection  process.  A  complete 
explanation  of  terms  will  not  be  attempted.  However,  AFR  70-15  contains 
all  terms  and  can  be  used  as  a  reference. 

1.  Source  Selection  Process.  The  formalized  process  employed 
in  competitive,  negotiated  procurements  for  programs.  The  source  selec¬ 
tion  process  is  designed  to  insure  an  impartial,  equitable,  and  economic 
evaluation,  and  comparative  analysis  of  competing  offerors'  proposals 
and  their  capabilities.  This  provides  an  objective  basis  for  the  SSA 

to  select  the  source(s)  which  best  satisfies  the  government's  require¬ 
ments  . 

2.  Request  for  Proposal  (RFP).  The  RFP  is  the  formal  docu¬ 
ment  used  to  invite  contractors  to  submit  proposals  for  required 


3 


supplies  and  services.  It  is  the  medium  by  which  potential  sources 
are  introduced  to  the  job  and  sets  the  stacje  for  the  proposals,  eval¬ 
uation,  and  contract  definitization  (ASD  Reserve  Project  Number  78-017- 
Bl,  1979:1). 

3.  Evaluation  Criteria.  The  standards  used  in  the  source 
selection  process  to  measure  proposal  acceptability  in  regard  to  the 
government  requirements  stated  in  the  Request  for  Proposal. 

4.  Source  Selection  Authority  (SSA).  I  he  official  designated 
to  direct  the  source  selection  process,  approve  the  source  selection 
plan,  select  the  source(s),  and  announce  the  contract  award . 

5.  Source  Selection  Advisory  Council  (SSAC).  A  group  of 
senior  military  or  government  civilian  personnel  designated  to  serve  as 
the  staff  and  advisors  to  the  SSA  during  the  source  selection  process. 
They  analyze,  for  the  SSA,  the  results  of  the  Source  Selection  Evalua¬ 
tion  Board  (SSE3). 

6.  SSAC  Analysis  Report.  A  formal  report  which  contains  the 
overall  analysis  of  the  SSAC  in  the  form  of  relative  strengths,  weak¬ 
nesses,  and  risks,  prices  or  costs,  overall  technical  effectiveness, 
risk  analysis,  offerors'  capabilities,  energy  effectiveness,  environ¬ 
mental  awareness,  negotiation  results,  and  other  appropriate  factors. 

7.  Source  Selection  Evaluation  Board  (SSEB).  A  group  of 
military  and/or  government  civilian  personnel,  representing  the  various 
functional  and  technical  disciplines  for  a  procurement  appointed  by 
the  SSAC  to  direct,  control,  evaluate  proposals,  and  develop  summary 
facts  and  findings. 
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8.  SSEB  Evaluation  Report.  This  report,  prepared  by  the 
SSEB,  contains  evaluation  standards,  detailed  narrative  assessments  of 
each  proposal  against  these  standards,  numerical  scores  when  used,  and 
summary  appraisals  of  significant  strengths,  weaknesses,  and  risks  of 
each  proposal.  It  forms  the  basis  of  the  SS/\C  analysis. 

9.  Source  Selection  Officer.  A  staff  officer  who  advises  the 
organization  commander  on  source  selection  policies  and  procedures.  He 
also  advises  the  Program  Manager,  SSA,  SSAC,  or  SSEB  on  how  to  properly 
conduct  a  course  selection. 

10.  Procurement  Evaluation  Panel  (Review  Board).  A  group  of 
highly  qualified  government  officials  that  review  Requests  for  Proposals 
and  other  documentation  for  selected  procurements  to  assure  that  exces¬ 
sive  or  nonessential  technical,  management,  and/or  procurement  related 
requirements  are  eliminated,  that  the  RFP  sets  forth  clearly  what  the 
government  plans  to  buy,  and  that  business  management  considerations  are 
assessed  (AFR  70-15,  1976:1-1  -  1-3). 

These  definitions  apply  to  major  source  selection  where  structured 
procedures  are  required.  AFR  70-15  permits  considerable  flexibility 
for  those  programs  falling  below  the  major  program  financial  thresholds. 
The  following  quote  emphasizes  this  fact.  "Policies  and  procedures  con¬ 
tained  in  this  regulation  are  sufficiently  flexible  to  accomodate  a 
wide  range  of  requirements.  They,  therefore,  may  be  used  as  a  guide 
to  formally  evaluate  competitive  proposals  and  to  select  sources  for 
other  programs/projects  below  the  dollar  thresholds  prescribed  above" 
(AFR  70-15,  1976:1-1). 
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Small  dollar  procurements  are  usually  involved  in  projects/programs 
in  AE,  and  are  also  typical  of  many  other  program  offices.  This  moti¬ 
vated  ASD  to  publish  ASD  Pamphlet  800-7.  Its  objective  is  to  "stream¬ 
line,  simplify,  and  improve  the  effectiveness  of  the  source  selection 
process"  (ASDP  800-7,  1978:1-2).  It  specifies  procedures  which  are  to 
be  employed  for  less  than  major  source  selections.  Although  less  for¬ 
malized,  small  source  selections  utilize  procedures  similar  to  those 
in  major  source  selections. 

The  Procurement  Evaluation  Panel  was  mentioned  in  the  definitions. 
As  authority  is  delegated  downward,  the  SSA  eventually  rests  with  the 
Procuring  Contracting  Officer  (PCO).  In  small  programs  typical  of  AE, 
the  PCO  assumes  the  responsibilities  of  the  Procurement  Evaluation 
Panel.  The  PCO  has  three  responsibilities:  (1)  control  all  communica¬ 
tion  between  the  government  and  offeror,  (2)  assure  that  the  basis  for 
fully  acceptable  contracts  are  negotiated  with  responsible  offerors, 
and  (3)  assure  sound  procurement  practices  are  adhered  to  throughout 
the  entire  source  selection  process  (AEOI  70-2,  1978:5). 

Organizational  arrangements  are  altered  as  stated  in  ASDP  800-7. 
Significant  changes  have  been  successfully  instituted  where  the  SSA 
has  been  delegated  to  the  division  commander.  The  SSAC  and  SSEB  are 
deleted.  Instead,  a  small  group  of  evaluators  are  carefully  selected 
to  participate  as  members  of  an  evaluation  body  designated  as  the 
Evaluation  Committee  (EC)  (ASDP  800-7,  1978:7).  ASDP  800-7  describes 
the  duties  of  these  evaluators.  "Evaluators  arc  responsible  to  the 
Chief  of  the  EC  and,  in  turn,  to  the  SSA  for  their  contributions  and 
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conduct  during  tho  course  of  the  preparation  of  the  RfP,  proposal 
evaluation,  and  completion  of  the  selection  process"  (A5DP  800-7, 
1978:7). 

As  the  collar  amounts  are  further  reduced,  the  EC  is  dispensed 
with.  The  following  quote,  from  a  Systems  Command  Supplement  to 
AFR  70-15,  explains  why.  "Formalized  source  selection  procedures  for 
procurements  with  a  projected  contract  dollar  value  below  $1  million  are 
discouraged.  Rather,  regular  ASPR  award  procedures  (3-805  and  4-106) 
are  encouraged  for  procurements  where  the  complexities  and  risks  do  not 
require  the  use  of  formalized  source  selection  ..procedures"  (AFR  70-15, 
AFSC  Sup  1,  1977:1).  An  attempt  will  be  made  to  examine  source  selec¬ 
tion  in  this  relaxed  atmosphere. 

In  this  thesis  an  examination  of  a  part  of  the  source  selection 
process  will  be  undertaken.  Program  managers  and  their  supervisors 
will  be  administered  an  exercise  requiring  them  to  evaluate  an  RFP. 

Each  proposal  will  have  been  rated  in  relation  to  five  evaluation 
criteria  when  presented  for  evaluation.  This  task  of  evaluating  a  pro¬ 
posal  is  representati ve  of  the  type  of  decision  required  of  program 
managers  and  supervisors  in  source  selection. 

A  typical  program/project  in  AE  might  be  below  $1  million.  Thus, 
the  theoretical  source  selection  in  this  research  will  be  of  such  a 
size;  a  $400,000  program  for  RDT&E.  The  SSA  will  be  the  Procuring 
Contracting  Officer  in  accordance  with  AE0I  70-2.  This  $400,000  figure 
is  below  the  $500,000  threshold  for  RDT&E.  Thus,  the  SSA  is  delegated 
to  the  lowest  possible  level  of  authority  within  AE.  Such  a  delegation 


often  occurs. 


In  Appendix  A,  a  list  of  source  selection  actions  is  provided  for 
information.  It  states  actions  required  when  using  or.  Evaluation  Com¬ 
mittee.  An  Evolution  Committee  will  not  be  required  for  the  theoretical 
$400,000  source  selection.  However,  even  when  not  using  an  Evaluation 
Committee,  it  is  desirable  for  the  program  manager  to  follow  as  closely 
as  possible  the  actions  which  the  committee  would  have  taken.  Maximum 
effort  went  into  design  of  the  formal  procedures  associated  with  major 
programs.  Past  errors  were  considered  when  constructing  the  formal 
regulations. 

Criteria  for  evaluation  of  proposals  are  stated  differently  in 
APR  70-15,  ASPR  3-501,  arid  A5PR  4-10G.4.  The  significant  factors, 
regardless  of  how  they  are  stated,  continually  recur.  The  following  five 
criteria  are  a  distillation  of  statements  in  APR  70-15  and  ASPR:  (1) 
technical,  (2)  operational,  (3)  logistics,  (4)  management,  and  (5)  cost. 
These  five  criteria  will  be  used  to  evaluate  source  selection  decision 
making. 

Statement  of  the  Problem 

There  are  five  organizational  units  within  AE  which  utilize  program 
managers.  These  are  Avionics  and  Aircraft  Accessories  Systems  Program 
Office  (SPO)-AEA,  Life  Support  SP0-AEL,  Reconnaissance/Strike  SP0-AER, 
and  Electronic  Warfare  SP0-AEW.  The  specific  intent  is  to  evaluate 
source  selection  decision  making  within  these  organizations.  In  addi¬ 
tion,  an  examination  and  comparison  of  decisions  by  program  managers  and 
their  immediate  supervisors  will  be  undertaken. 
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The  part  of  source  selection  to  be  examined  is  one  of  the  actions 
which  would  normally  be  performed  by  the  Evaluation  Committee;  eval¬ 
uating  proposals  which  have  already  been  rated  in  relation  to  specific 
evaluation  criteria.  In  a  less  than  major  source  selection,  the  prograr 
manager  or  Iris  direct  supervisor  could  be  responsible  for  accomplishing 
this  task.  By  choosing  to  evaluate  decision  making  in  less  than  major 
source  selections,  research  is  constrained  to  an  area  representative  of 
AE.  All  of  AE's  program  managers  are  familiar  with  this  type  of  source 
selection. 

Specific  hypotheses  to  be  tested  are: 

HI:  Evaluation  criteria  are  weighed  consistently  by 
personnel  within  AE. 

H2:  There  are  no  differences  among  AE  three  letter  groups 
with  respect  to  source  selection  decisions. 

H3:  There  is  no  difference  in  source  selection  decisions 
made  by  program  managers  and  their  direct  supervisors. 

H4:  Source  selection  decisions,  in  the  aggregate,  are 
highly  predictible  using  only  two  or  three  of  the 
five  evaluation  criteria. 

H5:  Program  managers  and  their  direct  supervisors  will  be 
able  to  assimilate  a  policy  prioritizing  evaluation 
criteria. 

HG:  Program  managers- and  their  immediate  supervisors  will 
assimilate  and  apply  a  policy  to  the  same  degree. 

H7:  Three  letter  groups  will  assimilate  a  policy  to  the 
same  degree. 
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H8 :  There  is  negligible  difference  in  the  amount  of 

variance  explained  by  linear  and  non-linear  models. 


Assumptions 

The  assumption  is  made  that  correct  or  representative  evaluation 
criteria  have  been  identified  prior  to  conducting  the  source  selection 
decision  making  exercise.  This  is  necessary  for  the  results  of  the 
analysis  to  be  meaningful. 

Limi tations 

There  are  approximately  122  program  managers,  30  of  which  are  super¬ 
visors.  An  attempt  will  be  made  to  administer  the  exercise  to  each  of 
them.  Thus,  full  coverage  or  representative  coverage  will  be  obtained 
for  a  specific  organizational  unit  within  Aeronautical  Systems  Division. 

In  the  next  chapter,  Methodology,  the  construction  of  an  exercise 
to  analyze  source  selection  is  discussed.  Then,  the  techniques  used  in 
evaluating  the  results  of  the  exercise  are  presented. 
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CHAPTER  II 


METHODOLOGY 

Modeling  the  Source  Selecti on  Deci s ion 

In  Chapter  I,  the  judgment  to  be  modeled  was  identified.  In  addi¬ 
tion,  five  evaluation  criteria,  henceforth  cues,  were  presented.  The 
next  problem  is  how  to  construct  an  exercise  using  these  cues  to  analyze 
a  source  selection  decision.  Multiple  linear  regression  is  a  very  power¬ 
ful  tool  which  can  be  used  to  accomplish  this  task. 

The  first  use  of  linear  models  was  proposed  by  Benjamin  franklin 
in  1807.  In  a  letter  to  his  friend,  Joseph  Priestly,  Franklin  advocated 
the  use  of  what  he  called  moral  or  prudential  rules  in  arriving  at  deci¬ 
sions.  lie  specified  how  to  apply  a  linear  technique  in  making  decisions 
(Dawes  &  Corrigan,  1974:95).  In  1923,  Henry  A.  Wallace  proposed  building 
a  linear  model  to  represent  a  corn  judge's  actions  (Dawes  &  Corrigan, 
1974:100).  Brunswik  (1940)  proposed  the  use  of  multiple  regression  as 
a  model  for  human  use  of  information.  Hoffman  proposed  that  linear 
models  could  be  used  to  represent  expert  judgment  (Hoffman,  1960:116). 

Since  the  time  of  Brunswik,  linear  analysis  has  been  applied  to 
depth  perception,  person  perception,  clinical  inference,  and  conflict 
resolution.  The  regression  technique  has  also  served  as  a  model  for 
examining  ability  to  detect  and  use  correlations  in  quantitative  pre¬ 
diction  tasks  (Beach,  1967:277).  Slovic.  and  Lichtenstein  cite  over  30 
studies  illustrating  the  tremendous  diversity  of  judgmental  tasks  to 
which  the  model  has  been  applied  (Slovic  &  Lichtenstein,  1971:677). 
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In  the  Air  Force,  multiple  regression  has  been  applied  in  a  variety 
of  situations.  The  difficulty  level  of  jobs  within  the  Medical  Materiel 
Career  Ladder  was  adequately  determined  using  a  three-variable  regres¬ 
sion  equation  (Mead,  1970: i i i ) .  Multiple  regression  was  also  used  to 
evaluate  the  job  difficulty  of  Accounting  and  Finance  supervisors  (Mead 
and  Christal,  1970:1).  Multiple  regression  was  also  used  at  the  United 
States  Air  Force  Academy  to  evaluate  and  improve  the  cadet  rating  system 
(Taylor,  1976:217).  It  has  also  been  used  in  several  AFIT  thesis  efforts 

Applicabil ity  of  the  linear  Model 

Bottenberg  and  Ward  (1963)  state  that: 

If  it  is  possible  to  obtain  all  the  information  available 
to  decision-makers  and  an  adequate  sample  of  their  deci¬ 
sions,  we  usually  can  formulate  a  regression  model  that 
satisfactorily  accounts  for  the  decisions. 

This  process  of  weighting  and  combining  information  (often  referred  to 

as  cues)  can  be  represented  by  an  equation  of  the  form 

Y  =  b i X !  +  b2X?.  +  .  .  .  +  biXi 

where  Y  is  the  judgment,  X.  is  the  value  of  the  ith  cue,  and  b^ 
represents  the  importance  of  the  ith  cue  (Slovic  and  Hoffman,  1969:2). 

The  b^  are  often  called  beta  weights. 

Even  if  it  were  possible  to  obtain  all  information  available  to 
the  decision  maker,  often  the  decision  is  made  in  the  face  of  uncer¬ 
tainty.  The  question  then  arises,  can  multiple  linear  regression  do 
an  adequate  job  under  these  circumstances.  Beach  (1967)  makes  the 
assumption  that  men  must  attempt  to  function  effectively  in  an  environ- 
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merit  about  which  they  receive  only  equivocal  information.  He  states 
that  when  faced  with  this  type  of  situation,  "Multiple  regression  is 
used  as  the  model  for  optimal  use  of  the  equivocal  information" 

(Beach,  1967:278).  In  further  support  of  multiple  linear  regression, 
Dawes  and  Corrigan  state  that,  "It  is  not  surprising  that  linear  models 
outperform  intuitive  judgment"  (Dawes  and  Corrigan,  1974:105).  Thus, 
the  literature  provides  a  strong  support  for  the  application  of  linear 
models  to  decision  making  processes.  This  model  has  been  used  in 
describing  decisions  or  as  a  substitute  for  the  decision  maker. 

The  Use  of  Cues 

Most  of  the  important  stimuli  about  which  judgments  are  made  are 
multidimensional.  A  cue,  information  bit,  component  attribute,  or 
dimension  characterizes  this  multidimensional  stimulus  (Slovic  and 
Hoffman,  1969:1).  Thus,  in  simple  language,  a  cue  is  a  piece  of  infor¬ 
mation  to  be  utilized  by  the  decision  maker. 

A  question  which  often' arises  is  how  many  cues  to  provide  the  dec! 
sion  maker.  Another  concomitant  question  is  how  many  levels  or  dimen¬ 
sions  to  provide  each  cue.  Slovic  and  Lichtenstein  warn  against  using 
too  many  cues.  In  their  study,  increased  numbers  of  cues  led  to  decrees 
accuracy.  The  following  quote  emphasizes  this: 

In  summary,  there  is  a  small  amount  of  evidence  that 
increasing  the  amount  of  information  available  to  the 
decision  maker  increases  his  confidence  without 
increasing  the  quality  of  his  decisions  and  makes  his 
decisions  more  difficult  to  predict.  (Slovic  and 
Lichtenstein,  1971:687) 
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Linear  vs.  Non-linear  Models 


Numerous  examples  citing  the  strength  of  the  linear  model  have  been 
presented.  At  this  point,  some  questions  about  non-linearity  will  be 
examined.  The  question  is:  Do  subjects  combine  cues  in  a  linear  or  non¬ 
linear  fashion?  Brehmer  states  there  is  little  evidence  that  humans 
use  information  is  any  other  than  the  linear,  additive  (or  averaging) 
one.  He  further  asserts  that  "Still,  the  majority  of  the  studies  indi¬ 
cate  that  most  or  all  the  variance  in  his  (the  subject's)  response  system 
can  be  accounted  for  by  a  linear  multiple-regression  equation"  (Brehmer, 
1969:491). 

Numerous  studies  have  uncovered  instances  of  interactions  among 
cues.  Even  when  these  interactions  arc  present,  Slovic  and  Lichtenstein 
assert  that  "The  linear  model  accounts  for  all  but  a  small  fraction  of 
predictable  variance  in  judgments  across  a  remarkably  diverse  spectrum 
of  tasks"  (Slovic  &  Lichtenstein,  1971:681).  Thus,  a  minimum  of  time 
will  be  spent  on  non-linear  considerations  in  this  research  effort. 


Relative  Heights 

Hoffamn  identified  three  limitations  with  using  beta  weights  in 
multiple  regression.  These  are  (1)  direct  comparisons  of  sets  of  betas 
between  subjects  is  not  meaningful,  (2)  beta  coefficients  do  not  account 
for  all  the  predictable  variance,  and  (3)  beta  coefficients  do  not  allow 
for  the  assessment  of  the  independent  contribution  of  each  predictor 
(Hoffman,  1960:120).  Therefore,  he  advocated  the  use  of  relative 
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Vi  here 


b.  =  the  beta  coeff  icient  for  tiu  i  th  predictor 

i  y 

r.  =  the  validity  coefficient  (correlation  with  judgment) 
"  of  the  i th  predictor 

R2  =  the  squared  multiple  conflation  coefficient  for 
y  subject  Y  (Hoffman,  1960:121). 


Slovic  and  Lichtenstein  point  out  that  "Since  the  sum  of  the  rela¬ 
tive  weights  is  1.0,  Hoffman's  index  describes  the  relative  contribution 
of  each  of  the  predictors  as  a  proportion  of  the  predictable  linear 
variance"  (Slovic  &  Lichtenstein,  1971:658). 

To  the  researcher  who  is  primarily  interested  in  relative  weights, 
orthogonal  designs  often  seem  preferable  to  designs  in  which  the  cues 
are  correlated  in  a  representati ve  manner.  "One  device  used  to  insure 
orthogonality  has  been  to  construct  stimuli  by  producing  factorial 
combinations  of  the  cues"  (Slovic  l  Lichtenstein,  1971:658).  Such  a 
design  was  employed  when  constructing  the  source  selection  exercise  for 
this  thesis.  With  this  design  r..  =  b^  so  the  equation  for  computing 

the  relative  weights  reduces  to 


r; 


Identification  of  the  P roblem 

Interviews,  with  three  letter  chiefs  within  AE,  were  conducted  to 
learn  more  about  problems  within  the  source  selection  process.  In  addi¬ 
tion,  all  the  source  selection  regulations  and  documents  utilized  by 
personnel  within  AE  were  obtained.  Five  evaluation  criteria  were 
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bach  cue  was 


identified  from  interviews  and  from  the  regulations, 
provided  two  levels;  adequate  or  excellent.  Lieutenant  Colonel  David 
L.  Wei  ton  (A LA)  offered  particularly  strong  support  on  all  phases  of 
this  stage  of  the  research.  He  assisted  in  identifying  and  clarifying 
the  cues.  Also,  he  provided  the  contacts  within  AC  necessary  to  obtain 
permission  to  conduct  the  source  selection  exercise. 

Appl i ca tio n  of  the  Cue s 

Tiie  next  task  was  to  construct  a  source  selection  exercise  utiliz¬ 
ing  the  evaluation  criteria.  The  intent  was  to  model  how  program  mana¬ 
gers  and  supervisors  evaluate  proposals  which  have  been  rated  in  relation 
to  the  evaluation  criteria.  Since  only  five  evaluation  criteria  were 
identified  a  full  factorial  design  was  employed  in  constructing  the 
exercise.  A  design  which  exhausts  all  possible  combinations  of  the  cues 
is  often  called  full  factorial.  Each  cue  was  coded  either  excellent  or 
adequate.  Thus,  the  full  factorial  design  required  2r',  or  32,  decisions 
to  exhaust  all  possible  combinations  of  the  cues.  The  exercise  is  pro¬ 
vided  for  examination  in  Appendix  B. 

In  a  Request  for  Proposal  (RFP),  evaluation  criteria  are  listed  in 
a  prioritized  manner.  The  specific  intent  is  for  offerors  to  incorporate 
this  priority  when  preparing  proposals.  One  measure  of  how  well  an  RFP 
met  objectives  is  the  degree  to  which  the  proposal  incorporates  the 
prioritized  critiera.  Program  managers  are  often  involved  in  examining 
this  facet  of  a  proposal. 

In  the  first  part  of  the  exercise  demographic  questions  were  asked. 
The  second  part  of  the  exercise  required  that  the  32  decisions  be  made 
with  no  stated  priority.  Thus,  the  intent  was  to  capture  the  policy  of 


1G 


each  individual  with  no  guidance.  In  the  third  part  of  the  exercise, 
a  priority  of  the  cues  was  provided.  Then,  each  individual  was  asked 
to  evaluate  32  additional  proposals.  This  allowed  comparison  of  each 
individual's  decisions  with  and  without  a  policy. 

To  insure  that  each  individual  would  not  be  able  to  recognise  a 
pattern  between  the  same  32  decisions  in  parts  two  and  three,  a  random 
number  table  was  used  to  determine  the  order  of  the  32  combinations  of 
the  cues.  A  different  random  number  generator  was  used  for  parts  two 
and  three  of  the  exercise. 

The  appropriate  linear  model  for  this  study  is 

Y  =  biXi  +  b2X2  +  b  3  X3  +  bi,  X4  +  b^Xs 

where  Xi  through  X5  are  the  evaluation  criteria,  as  follows: 

X 1  =  Technical 

X2  =  Operational 
X3  ~  Logistics 
X4  =  Management 
X5  =  Cost. 

In  Appendix  B  each  of  the  evaluation  criteria  is  defined. 

Each  of  the  32  proposals  can  bt  represented  mathematically  by 
orthogonal  vectors.  This  is  presented  in  Appendix  C.  Orthogonality 
assures  that  the  evaluation  criteria  will  be  uncorrelated  in  the  regres¬ 
sion.  Thus,  the  usefulness  of  the  evaluation  criteria  in  arriving  at 
the  proposal  evaluation  is  strictly  measured  by  the  intercorrelation 
between  each  evaluation  criteria  and  the  decision  (Darlington,  1962: 
162).  Each  of  the  32  evaluations  made  by  the  experimental  subject  is 
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strictly  characterized  by  the  numerical  value  (0,  1,  .  .  .  ,  10)  of 
his  evaluation  and  one  of  32  orthogonal  predictor  vectors,  each  of 
which  represents  a  unique  proposal  state.  It  is  thus  simple  to  deter¬ 
mine  the  importance  the  subject  places  on  each  of  the  five  evaluation 
criteria  with  and  without  a  policy. 

Significance  of  R2 

R2,  the  squared  multiple  correlation  coefficient  for  the  regression, 
provides  information  as  to  how  well  an  individual's  decision  process  was 
modeled.  It  is  one  of  the  printed  outputs  of  Lieutenant  Colonel  Charles 
W.  McNi chols1  computer  program.  The  following  quote  defines  R:  "The 
multiple  correlation  (R)  provides  a  measure  of  control  because  it  mea¬ 
sures  the  correspondence  between  an  individual's  judgments  and  predictions 
from  a  specific  model"  (Hammond, et  al ,  1977:279).  Jones,  et  al  (1976) 
provide  an  F-test  to  evaluate  whether  or  not  R  is  statistically  signi¬ 
ficant.  For  this  test, 

Fo  =  (n-k-l)R2 

k( 1 -R2) 

where  the  terms  mean  the  following: 

n  =  the  number  of  decisions  (32) 

k  =  the  number  of  cues  (5) 

R2  =  the  squared  multiple  correlation  coefficient.  F  is 
distributed  with  p  and  n-p-1  degress  of  freedom 

This  test  was  used  to  evaluate  an  individual's  R  in  order  to  determine 

whether  or  not  any  individual's  model  was  statistically  valid. 
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Pro- tost 


In  order  to  assess  the  effectiveness  of  the  source  selection 
exercise,  a  pre-test  was  conducted.  Five  subjects,  one  each  from  the 
five  divisions  within  AE,  participated.  Four  subjects  completed  all 
parts  of  the  exercise;  with  and  without  a  policy.  Within  one  day  after 
completing  the  exorcise,  each  subject  was  asked  to  make  recommendations 
for  improvements.  None  of  the  subjects  had  any  difficulty  in  under¬ 
standing  the  directions  and  accomplishing  the  exercise.  One  recommen¬ 
dation  which  was  incorporated  was  a  change  to  the  size  of  the  theoretical 
proposal.  It  was  changed  from  $10,000  in  the  pre-test  to  $400,000  in  the 
actual  exercise. 

Pre-Test  R^su Its 

Four  of  the  five  people  who  too!:  the  pre-test  completed  all  sections 
of  it.  In  Table  II  the  individual  and  average  of  the  individual  R2ls  for 
the  four  are  presented. 


Case 

TABLE  II 

Pre-Test  R2,s 

R2  -  No  Policy  R2  - 

_With  Policy 

1 

.830 

.855 

2 

.466 

.837 

3 

.550 

.515 

4 

.930 

.957 

Average 

.707 

.794 

An  F-test  to 

tost  the  significance  of  R2 

was  mention! 
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At  the  five  percent  significance  level  an  F-value  was  extracted  from 
tables  (F  =  2. 59).  One  can  then  solve  for  the  R2  value  which  would 
have  caused  rejection  of  this  statistic,  R2  =  .32?.  None  of  the  indi¬ 
viduals  and  none  of  the  average  R2  values  fall  below  this  figure. 

A  comment  about  responses  received  during  the  interviews  after 
the  exercise  is  now  appropriate.  The  overall  exercise  did  make  sense 
to  the  test  subjects.  They  understood  what  they  were  asked  to  do.  The 
instructions  were  adequate  and  understandable.  The  definition  of  each 
evaluation  criteria  was  clear.  In  their  opinion,  the  evaluation  cri¬ 
teria  were  typical  of  source  selections  within  AE.  There  are  not  any 
other  evaluation  criteria  which  should  be  included  or  were  thought  to 
be  more  relevant  than  those  which  were  used.  It  was  not  felt  that  an 
example  was  necessary  in  the  instructions.  Last,  the  decision  that  the 
subjects  were  asked  to  make  was  meaningful  as  compared  with  the  overall 
kind  of  judgment  in  source  selection. 

The  interviews  after  the  pre-test  uncovered  a  possible  problem  with 
the  completion  of  the  source  selection  exercise.  A  wide  diversity  of 
levels  of  experience  exists  within  AE.  Some  program  managers  have 
never  accomplished  a  source  selection  while  others  have  accomplished 
many.  However,  all  the  program  managers  are  somewhat  familiar  with  the 
process.  Because  of  the  wide  degree  of  difference  in  experience,  two 
additional  demographic  questions  were  added  to  the  exercise:  (1)  Have 
you  ever  been  involved  in  an  actual  source  selection?  and  (2)  If  so, 
about  how  many? 
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Adm i  ni  stra t iono f  the  Uxor cis e 

After  incorporating  acceptable  recommendations ,  the  exercise  was 
distributed  to  9b  personnel  within  AE  on  9  June  1979.  The  five  parti¬ 
cipants  in  the  pre-test  were  excluded.  Because  the  initial  response 
rate  was  low,  a  letter  extending  the  cutoff  date  was  distributed.  The 
letter  was  written  only  after  it  was  determined  that  leave,  excessive 
TOY,  and  heavy  June  workloads  had  interferred  with  completing  the 
exercise.  A  copy  of  this  letter  is  presented  in  Appendix  D  for  refer¬ 
ence. 

Codi ng  of  the  Data 

The  data  was  coded  to  fit  the  format  of  a  FORTRAN  program  written 
by  Lieutenant  Colonel  McNichols,  AFIT  Systems  Management  Department. 

This  program  computed  relative  weights  and  coded  this  output,  along 
with  each  individual's  demographic  data,  onto  disc.  This  coding  to 
disc  was  necessary  in  order  to  have  individual  data  available  for 
regression  analysis  using  the  Statistical  Package  for  the  Social 
Sciences  (SPSS). 

Analysis  of  the  Experi menta  1_ _Da ta_ 

The  Aeronautical  Systems  Division  (ASD)  CDC  6600  computer  was  used 
for  the  data  analysis.  It  employed  the  SPSS  multiple  regression  algorithm 
( Mi e ,  et  al ,  1975).  Also,  Lieutenant  Colonel  McNichols'  program  was 
employed  to  compute  individual,  relative  weights  with  and  without  a  policy. 

The  SPSS  capabilities  are  numerous.  A  total  of  thirty  seperate 
regression  models  were  generated  for  this  study.  These  models  represent 
the  following  groups:  (1)  program  managers  with  and  without  a  policy, 
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(2)  supervisors  with  and  without  a  policy,  (3)  dividions  one  through 
five  (1-AEC,  2-AEW,  3-AER,  4-AEA,  5-AEL)  with  arid  without  a  policy, 

(4)  interaction  models  for  the  above  groupings,  and  (5)  source  selec¬ 
tion  experience  versus  no  source  selection  experience  with  and  without 
policy.  Generation  of  these  models  was  quite  easy  using  the  Select  If 
and  Compute  features  of  SPSS  ( N i e ,  et  al ,  1975). 

Testing  for  Statistical  Differences  Between  Groups 

Each  of  the  models  was  tested  for  differences  among  the  groups.  The 
groups  examined  were  divisions,  program  managers  versus  supervisors,  and 
source  selection  experience  versus  no  source  selection  experience.  A 
statistically  significant  difference  would  indicate  that  the  regression 
coefficients  represent  different  populations.  Therefore,  the  importance 
given  to  the  evaluation  criteria  by  at  least  one  group  in  the  comparison 
is  significantly  different  from  the  others.  To  accomplish  this,  the 
F-test  described  by  Gregory  C.  Chow  was  used  (Chow,  1960:599). 

The  null  hypothesis  for  the  Chow  test  is  that  there  is  no  difference 
in  the  regression  coefficients  (b/s)  for  the  groups  involved.  The 
alternate  hypothesis  is  that  the  regression  coefficients  are  different 
for  at  least  one  pair  of  the  groups.  This  is  equivalent  io  saying  that 
at  least  two  of  the  groups  utilize  evaluation  criteria  differently  in 
evaluating  proposals.  The  level  of  significance  for  all  F-tests  in  this 
study  was  .05.  The  method  of  calculating  this  F-test  for  groups  is  shown 
in  Appendix  E. 

Deletion^of  Cues 

A  question  often  asked  is,  does  the  decision  maker  utilize  all 
information  available  to  him.  lo  investigate  information  utilization 
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in  source  selection,  cues  were  examined  for  their  contribution  to  the 
rating  a  proposal  received.  In  order  to  determine  whether  01  not  a 
cue  could  be  deleted,  a  somewhat  arbitrary  90  percent  floor  was  esta¬ 
blished  for  contribution  to  the  overall  decision.  In  other  words,  the 
relative  weights  were  examined  for  magnitude.  Any  cue  or  combination  of 
cues  contributing  less  than  ten  percent  to  the  overall  decision  were 
dropped.  The  overall  group,  supervisors,  program  managers,  all  divisions, 
and  individuals  were  examined  with  and  without  a  policy. 


As si mi  1 ati on  o f  a  Pol  icy 

To  determine  whether  or  not  the  policy  was  assimilated,  Spearman's 
Rho  was  utilized  (Mendenhall  and  Sheaffer,  1973:549).  Lieutenant  Colonel 
McNichols  wrote  a  FORTRAN  program  to  calculate  this  value.  The  groups 
examined  for  policy  assimilation  were  program  managers,  supervisors,  and 
divisions. 

Spearman’s  Rho  is  used  to  test  for  correlation  bewteen  two  paired 
variables.  The  Rho,  or  rank  correlation  coefficient,  tests  an  associa¬ 
tion  between  two  populations.  Rho  (R  )  is  calculated  by  the  following 
equation: 

6  l  (Kry.i)2 

r#  -  1  - 

s  n( n2-l ) 


where, 

x.  =  the  priority  (or  rank  order)  associated  with  an  evalua- 
1  tion  criteria  as  a  part  of  the  guidance 

y.  =  the  priority  (or  rank  order)  given  to  an  evaluation 
1  criteria  by  a  subject 

n  =  the  number  of  cues  (Mendenhall  and  Schaeffer,  1973:550). 
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Spearman's  Rho  for  each  subject  was  calculated  for  parts  two  and 
three  of  the  exercise.  In  part  two,  the  subjects  were  given  rio  priority. 


Thus,  a  low  r  would  be  expected.  In  part  three,  a  priority  was  given. 
Thus,  a  higher  value  of  r  would  be  expected. 

A  paired  t-test  was  conducted  for  each  of  the  groups  mentioned 
above.  The  results  indicate  whether  groups  could  assimilate  a  policy. 
Specific  hypotheses  tested  were 

V  'V'z  * 0 


where, 

p.  =  the  population  mean  Spearman  Rho  value  for  a  group 
with  no  policy 

P2  =  the  population  mean  Spearman  Rho  value  for  a  group 
with  policy. 


The  t-statistic  is  calculated  as  follows: 


t  = 


d 


//n" 


where, 

x,.  =  the  Spearman  Rho  value  for  individual  i  with  no 
policy 

x? •  =  the  Spearman  Rho  value  for  individual  i  with 
policy 


n  =  the  number  of  subjects  within  a  group 


d 
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'  (d-d)2 

s^2  =  l  — ri- i-~  (Mendenhall  and  Sheaffer,  1973:446). 

i=l 

t  has  n-1  (57)  degrees  of  freedom. 

Test  for  Difference  in  Ability  to  Assimilate  a  Policy 

To  test  for  differences  in  ability  to  assimilate  a  policy  among 
groups,  a  t-test  to  measure  differences  in  group  means  was  employed. 
Mean  Spearman  Rho  values  for  groups  (with  policy)  were  compared.  The 
hypotheses  tested  were: 

V  "l  '  "2 

V  >'l  *  >‘2 

where, 

p 1  =  the  population  mean  Spearman  Rho  value  for  group  one 

y^  =  the  population  mean  Spearman  Rho  value  for  group  two. 

The  t-statistic  has  the  following  form: 

t  Tl  " 

where, 

Yj  =  the  sample  mean  Spearman  Rho  value  for  group  one 

Y2  =  the  sample  mean  Spearman  Rho  value  for  group  two 

m  =  the  number  of  subjects  in  group  one 
n2  -  the  number  of  subjects  in  group  two 
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(s)?  and  s?2  are  the  sample 


s?  =  (ru-1 ) is i2  _  +  (n?-l  )_S2 

n^  +  n2  -  2 

variances  for  groups  one  and  two.) 

t  has  (ih  +  n2  -  2)  degrees  of  freedom  (Mendenhall  and  Sheaffer,  1973:346). 
Tests  for  Non-Linearity 

In  order  to  test  for  non-linearity,  interaction  models  were  built 
for  the  following  groups:  program  managers,  supervisors,  divisions,  and 
the  overall  group;  with  and  without  a  policy.  The  interaction  model  was 
constructed  as  described  by  Lieutenant  Colonel  McNichols  (1978:4-35). 

A  two-way  interaction  model  was  built.  It  was  similar  to  the  fol 1  owing 
equati on: 


Yi  =  Bq  +  B i X i i  +  BzXi2  +  B3  Xi 1 Xi 2 

All  possible  combinations  of  two-way  terms  were  utilized.  The  original 
five  evaluation  criteria  were  included  in  the  model.  Then,  two-way 
interactive  terms  which  will  contribute  most  to  the  model  were  chosen 
and  then  included  in  a  regression.  This  is  often  called  forcing  in 
variables.  For  each  of  the  groups,  the  increase  in  R2  was  then  examined. 
In  the  next  chapter  results  are  presented.  Analysis  of  demographic 
results  is  presented  first.  Then,  analysis  of  hypotheses  follows. 
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CHAPTLR  III 


RESULTS 

Res  nits  of  SPSS  Descri  ptive  _S_ta  tisti  cs 

As  mentioned  in  Chapter  II,  for  convenience  the  outputs  of  Lieutenant 
Colonel  McNichols1  FORTRAN  program  were  coded  to  disc.  This  facilitated 
use  of  the  data  with  SPSS  programs.  An  initial  examination  of  the  data 
was  conducted.  Specific  SPSS  capabilities  employed  were  FREQUENCIES  and 
CROSSTABS.  FREQUENCIES  displays  how  the  data  was  grouped.  CROSSTABS 
checks  the  data  groups  (demographic  groups)  for  independence  (Nie,  ct  al , 

197b) . 

Results  of  FREQUENCIES  will  be  presented  first.  The  desire  is  to 
determine  the  basic  distributional  characteristics  of  each  of  the  demo¬ 
graphic  variables  to  be  used  in  the  subsequent  statistical  analysis-.  -  -  ,  -<• 

The  variables  run  wore  frequencies  of  responses  (proposals  rated)  for 
demographic  descriptors.  The  descriptors,  as  listed  in  Appendix  B.  were: 
organization,  supervisor  or  non-supervisor,  and  source  selection  versus 
no  source  selection  experience.  Results  for  organizations  arc  presented 
in  Table  III.  Results  for  supervisor  versus  non-supervisor  are  presented 
in  Table  IV.  Results  of  source  selection  versus  no  source  selection 

experience  are  presented  in  Table  V. 

Second,  CROSSTABS  was  run’  for  individual  responses  with  and  without 
a  policy.  One  of  the  outputs  of  CROSSTABS  is  Chi-square,  which  deter¬ 
mines  whether  or  not  a  systematic  relationship  exists  between  the  varia¬ 
bles.  Since  Chi-square  increases  as  the  number  of  cases  increases,  a 
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TABLE  III 


Distribution  of  Responses  by  Orejani/ation 


Organize! i on 

Absolute  Frequency 
of  Responses 

Relative 

Frequency 

(Percent) 

AEG 

6 

10.3 

A  EK1 

16 

27.6 

ACR 

13 

22.  it 

AEA 

13 

22  A 

AEL 

10 

17.2 

Totals 

58 

100.0 
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TABLE  IV 


Distribution  of  Responses  by  Level  of  Responsibility 


Relative 

Absolute  Frequency 

Frequency 

Status 

of  Responses 

(Percent) 

Non-Supervi sors 

/II 

70.7 

Supervi sors 

17 

29.3 

Total s 


58 


100.0 


TABLE  V 


Distribution  of  Responses  by  Source  Selection  Experience 


Experience 

Absolute  Frequency 
of  Responses 

Relative 

T requency 
(Percent) 

Source 

Selection 

42 

72.4 

No  Source 

Selection 

16 

27.6 

To  tctls 

58 

100.0 

further  statistic.  Phi  ($)  is  often  employed  to  correct  for  this. 
Phi  has  the  form 


1. 


where 

\(j2  =  Chi-square 

N  =  the  number  of  sample  cases. 

"Phi  takes  on  a  value  of  0  when  no  relationship  exists,  and  the  value 
of  +1  when  the  variables  are  perfectly  related.  .  ."  (Nie,  el  al,  197b: 
224).  It  should  be  notec  that,  while  Chi-square  might  be  statistically 
significant  (at  the  five  percent  level),  the  corresponding  value  of  <J> 
might  be  so  low  as  to  indicate  little,  or  a  negligible,  relationship. 

In  Table  ''I  these  calculations  are  presented  for  groups  with  and  without 
policy.  In  Table  VII  results  for  pairings  of  demographic  groups  are 
presented.  The  strongest  relationship  exists  for  R2  versus  organization. 
The  weakest  relationship  exists  for  R2  versus  supervisory  level.  With¬ 
out  a  policy,  R2  is  independent  of  supervisory  level.  None  of  the  Phi 
values  is  close  to  +1  so  a  perfect  relationship  does  not  exist  for  any 
of  the  comparisons  in  the  table.  Each  subsequent  section  will  deal  with 
specific  hypotheses.  Hypotheses  will  be  stated  before  presentation  of 
data  applicable  to  them. 

Relative  Weights 

Relative  weights  were  computed  for  individual  subject  and  groups. 

The  individual  relative  weights  are  presented  in  Appendix  F  for  refer¬ 
ence.  Relative  weights  with  and  without  a  policy  are  displayed.  The 


TABLE  VI 


Tests  for  Independent •.* 

R?  Versus  Demographic  Group 


(K'ithout  Policy) 


Level  of 


Compnri son 

Chi  --Square 

Significance 

Phi 

R  versus 

Organi zat ion 

262.10 

.00 

.376 

R  versus 

Supervi sory 
Level 

.39 

.53 

-- 

R  versus 
Source 
Selection 
Experience 

5.55 

.02 

O 

LO 

o 

(With  Policy) 

Level  of 


Compa r is on 

Chi-Square 

Significance 

Phi 

2 

R  versus 
Organization 

n2 

R  versus 

190.86 

.00 

.321 

Supervisory 

Level 

123.53 

.00 

.258 

2 

R  versus 

Source 

Selection 

Experience 

64.22 

.00 

.186 

TABLE  VII 


Tests  Tor  Independence  Between  Denog  rapin' c  Group 


Level  of 

Compari son 

Chi -Squa re 

Sign  i  f icance 

Organization 

versus 

Supervisory 

Level 

Supervisory 
Level  versus 

76.52 

.00 

Source 

Selection 

Experience 

66.54 

.00 

CO 

<T* 

in 

CO 

CO 

LO 

LO 

LO 

CO 

CO 

i — 

CO 

cr> 

i — 

LO 

LO 

CO 

LO 

CO 

LO 

r-- 

CO 

f— 

CO 

O 

, — 

CVI 

o 

LO 

CO 

v4* 

vj- 

CO 

LO 

LO 

*d* 

•d- 

CO 

LO 

CVJ 

CO 

LO 

LO 

+->  cr»  r>.  co  lo  lo  vi-  lo  < —  co  o  f'"'  o  < —  lo 

l/)  CP»  r^<  i —  LQ  CO  COCOCMCOLOr^CTJr—  CO  CO  LO 

o  i —  I —  CVI  r—  «—  *—  r—  CMr—r—  r—  r—  CO  r—  r~r— 

U  •  •  •  •  •  . 


*A  =  No  Policy 
**B  =  With  Policy 


hand-calculated  relative  weights  are  presented  in  Table  VIII.  The 
group  relative  weights  ore  not  an  average  of  individual  relative  weights. 
Rather,  all  group  decisions  were  included  in  constructing  a  regression 
model  for  the  group. 

Consistency  of  Responses 

Hi:  Evaluation  criteria  are  weighed  consistently 
by  personnel  within  AE. 

R2  measures  how  consistently  individuals  weighed  evaluation  criteria. 
A  statistical  test  for  R2  was  presented  under  the  section  dealing  with 
pre-test.  Individual  R2,s  are  presented  in  Appendix  F.  As  presented 
there.  Case  48A  was  the  only  one  which  was  insignificant  from  a  statis¬ 
tical  point  of  view.  It  fell  below  the  minimum  R2  requirement  of  .322. 

In  addition  to  looking  at  individuals,  the  average  of  all  individuals  and 
the  overall  group  RZls  are  presented  in  Table  IX. 

None  of  the  figures  in  Table  IX  fall  below  the  computed  R2  figure 
of  .322.  Thus  personnel  within  AE  appeared  to  be  internally  consistent 
in  weighing  evaluation  criteria.  Hypothesisone  cannotbe  rejected.  The 
lower  R2  for  the  overall  group  indicates  that  diverse  policies  exist  in 
how  individuals  utilize  evaluation  criteria. 

Measuring  Differences  in  Groups 

H2:  There  are  no  differences  among  AE  three  letter 

groups  with  respect  to  source  selection  decisions. 

Hs:  There  is  no  difference  in  source  selection  deci¬ 
sions  made  by  program  managers  and  their  direct 
supervisors . 

Chow's  F-test  was  used  to  test  hypotheses  two  and  three.  For  a 
hypothesis  to  be  rejected,  ,  the  computed  F-value,  must  be  greater 


TABLE  IX 


Average  of  Individuals  and  Overall  Group  R2's 


Overall 

R 2 -  No  Po 1 i c v 

R2  -  Hi  th  Po  1  i  cy 

Group 

—  .  — — - *L. 

Regression 

.363 

.454 

Average  of 

Individual 

Regressions 

.820 

.855 
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than  F  ,  the  F-value  from  statistical  tables.  F  is  determined  by  the 
level  of  significance  (.06),  the  numerator  degrees  of  freedom,  and  the 
denominator  degrees  of  freedom.  The  results  are  shown  in  Tabic  X.  The 
results  would  indicate  rejection  of  If?.;  there  are  differences  in  three 
letter  groups  in  AE  with  respect  to  source  selection.  The  results  sup¬ 
port  H3;  there  is  no  difference  in  source  selection  decisions  made  by 
program  managers  and  their  direct  supervisors.  The  results  of  an  addi¬ 
tional  test  not  related  to  a  specific  hypothesis  are  also  presented 
at  the  bottom  of  Table  X.  A  test  for  differences  in  source  selection 
versus  no  source  selection  experience  was  conducted.  There  was  a  sig¬ 
nificant  difference  in  the  decision  making  of  those  who  had  participated 
in  source  selection  and  those  who  had  not. 

Delet ion  0 f  Cues 

H>*:  Source  selection  decisions,  in  the  aggregate, 
are  highly  predictable  using  only  two  or  three 
of  the  evaluation  criteria. 

To  examine  this  hypothesis  it  was  decided  that  any  cue  or  combina¬ 
tion  of  the  cues,  which  accounted  for  less  than  ten  percent  of  the  deci¬ 
sions,  was  considered  to  not  contribute  sufficiently  to  the  rating  a 
proposal  received.  Group  results  will  bo  presented  first.  Then,  results 
for  individuals  will  be  presented.  Cues  were  inspected  by  contribution 
to  relative  weights.  In  Table  XI  group  results  are  presented.  In  addi¬ 
tion  to  these  results,  individuals  were  also  examined.  For  individuals 
there  was  no  consistent  pattern  to  the  cues  which  could  be  dropped.  So, 
percentages  were  computed  for  individuals.  These  percentages  represent 
the  number  of  subjects  which  can  be  represented  with  three  cues  or  less. 
For  these  subjects,  the  cue  or  combination  of  cues  deleted  accounted 
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TABU  X 


Results  of  Chow's  F  Statistic  Testing  for 
Differences  in  Group's  Decision  Models 


H2:  No  Differences  Among  Three  Letter  Groups  (Without  Policy) 


Comparison  Groups 

F 

_ o_ 

[\_05 

Reject  H;>? 

AEG/AEW/AER/AEA/AEL 

5.15 

1 .52 

YES 

AEG/AEW 

3.44 

1.75 

YES 

AF.G/AER 

4.23 

1  .75 

YES 

AEG/AEA 

6.10 

1.75 

YES 

AEG/AEL 

3.30 

1.75 

YES 

AEW/AER 

.93 

1  .75 

NO 

AEW/ACA 

3.09 

1 .75 

YES 

AEW/AEL 

.21 

1.75 

NO 

AER/AEA 

2.18 

1.75 

YES 

AER/AEL 

1.16 

1  .75 

NO  ■ 

AEA/AEL 

2.84 

1  .75 

YES 

No  Differences  Among 

Th  ree 

Letter  Groups 

(With  Pol 

AEG/AEW/AER/AEA/AEL 

3.22 

1 .52 

YES 

AEG/AEW 

.78 

1 .75 

NO 

AEG/AER 

2.24 

1.75 

YES 

AEG/AEA 

1 .52 

1.75 

NO 

AEG/AEL 

3.30 

1 .75 

YES 

AEW/AER 

.65 

1.75 

NO 

AEW/AEA 

1.39 

1.75 

NO 

AEW/AEL 

1 .23 

1 .75 

NO 

AER/AEA 

2.62 

1 .75 

YES 

AER/AEL 

.26 

1 .75 

NO 

AEA/AEL 

3.37 

1 .75 

YES 

'icy) 


38 


TABLE  X  (Continued) 


H3:  No  Differences  Between  Supervisory  Levels 


Comparison  Groups 

F 

o_ 

F.05 

Reject  H ? 

Supervisors/ 

Program  Managers 
(Without  Policy) 

1.23 

2.10 

NO 

Supervisors/ 

Program  Managers 
(With  Policy) 

.94 

2.10 

NO 

Test  for  Differences  in 
Source  Selection  Experience 

Source  Selection/ 

No  Source  Selection 
(Without  Policy) 

16.35 

2.10 

YES 

Source  Selection 

No  Source  Selection 
(With  Policy) 

3.25 

2.10 

YES 
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TABLE  XI 

Group  Results-Cues  Which  Can  Ce  Dropped 


Wo  Policy 


Groiij) 

Doletable  Cues 

Overall  Group 

None 

Supervi sors 

None 

Program  Managers 

None 

AEG 

None 

AEW 

None 

AER 

None 

AEA 

Management 

AEL 

None 

Wi  th 

Pol  icy 

Overall  Group 

Logistics, 

Management 

Supervisors 

Logistics , 

Management 

Program  Managers 

Logistics, 

Management 

AEG 

Logistics, 

Management 

AEW 

Logistics, 

Management 

AER 

Logistics, 

Management 

AEA 

Logistics, 

Management 

AEL 

Logistics, 

Management 
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for  ten  percent  or  less  of  the  individual's  decision.  With  no  policy 
14  of  58,  or  24  percent  of  the  subjects  could  be  represented  by  three 
cues  or  less.  With  a  policy,  36  of  58,  or  62  percent  of  the  subjects 
could  be  represented  by  three  cues  or  loss. 

So,  the  results  relating  to  Ik  are  split.  With  no  policy,  it  cannot 
be  said  that  individuals  or  groups  can  be  represented  by  tv.  or  three 
cues.  With  policy,  individuals  and  group,  can  bo  represented  using  three 
cues . 

Ability  to  Assimilate  a  Pol i cy 

H s :  Program  managers  and  their  direct  supervisors 
will  be  able  to  assimilate  a  policy  prioritizing 
evaluation  criteria. 

Spearman's  Rho  was  calculated  to  evaluate  Hs .  The  Rho  value  indi¬ 
cates  how  close  a  group  was  to  the  policy.  Degree  of  conformity  to  the 
policy  was  measured  before  and  after  a  policy  was  given.  In  Table  XIII 
mean  values  of  this  Rho  are  presented  and  the  results  of  a  paired  t-test 
on  groups.  The  paired  t-test  indicates  whether  or  not  a  group  changed 
significantly  to  conform  to  the  policy.  The  t-statistic  indicates  a 
significant  change  toward  the  policy  in  all  groups.  Hs  is  supported; 
program  managers  and  their  direct  supervisors  were  able  to  assimilate 
a  policy  prioritizing  evaluation  criteria. 

Degree  of  Policy  As si  mil  a  tion 

H$:  Program  managers  and  their  immediate  supervisors 
will  assimilate  and  apply  a  policy  to  the  same 
degree . 

H7:  Three  letter  groups  will  assimilate  a  policy  to 
the  same  degree. 


Group  means  of  the  Spearman  Rho  values  (with  policy)  were  pre¬ 
sented  in  Table  XII.  In  Table  XIII  the  results  of  a  t-test  to  test 
for  differences  in  group  means  is  presented.  The  t-test  indicates 
whether  or  not  there  is  a  significant  difference  in  how  two  groups 
applied  a  policy.  The  means  tested  were  computed  from  Spearman  Rho 
values  with  policy. 

Ik  and  II?  cannot  be  rejected.  There  is  no  significant  difference 
in  how  any  of  the  groups  assimilated  the  policy. 

In  addition  to  the  t-tests  on  groups,  individual's  relative  weights 
were  inspected  for  conformance  to  the  policy.  Individual  relative 
weights  were  first  inspected  for  exact  conformance  to  the  policy.  Then, 
various  combinations  of  relative  weights  were  allowed  tc  shift  up  or 
down.  The  total  effect  cn  relative  weights  was  not  allowed  to  exceed 
.05.  Thus,  if  the  relative  weight  for  one  cue  went  up  .05,  the  relative 
weight  for  another  cue  or  combination  of  cues  must  shift  down  .05.  The 
results  are  presented  in  Table  XIV. 

The  individual  results  indicate  an  approximately  70  percent  degree 
of  conformance  or  near  conformance  to  the  policy. 

Tests  for  Non-Li nea ri ty 

He:  There  is  negligible  difference  in  the  amount  of 
variance  explained  by  linear  and  non-linear 
models . 

In  Table  XV,  R2  with  and  without  inclusion  of  interaction  terms  is 
presented.  Increase  in  R2,  due  to  incorporating  interaction  terms,  is 
indicated . 

Increases  when  incorporating  interaction  terms  are  meager  at  best. 
Ho  is  strongly  supported.  There  is  negligible  increase  in  the  amount 
of  variance  explained  by  the  non-linear  models. 
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TABLE  XII 


Group  Means  and  Paired  t-Tests 


Group 

Me an- No 
Policy 

Mean -With 

Pol  i  cy 

*0 

Significance 
_  Level_ 

t 

a 

Reject  Hr?* 

Supervisors 

.42 

.87 

-3.79 

.001 

-1  .645 

NO 

Program 

Managers 

.2(3 

.79 

-8.08 

.000 

-1 .645 

NO 

AEG 

.23 

.89 

-3.14 

.013 

-1 .645 

NO 

AEW 

.28 

.80 

-4.84 

.000 

-1 .645 

NO 

AER 

.23 

.88 

-4 .60 

.001 

-1  .645 

NO 

AEA 

.38. 

.71 

-3.56 

.002 

-1 .645 

NO 

AEL 

.51 

.85 

-3.99 

.002 

-1  .645 

NO 

*Hs:  Program 

Managers 

and  their  dii 

■ect  supervisors  will  I 

)e  able  to 

assimilate 

a  policy  prioritizing  evaluation  criteria. 
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TABLE  XII 1 


t~Tes 

Comparison  Groups 

t  of  Group  Means  to  Identify  Differences 
in  Ability  to  Assimilate  a  Policy 

,  Significance 

o  Level  ^.05  Reject  H( 

Supervisors/ 

Program  Managers 

-1  .11 

.272 

-1  .645 

HO 

AEG/AEW 

.86 

.472 

1.729 

NO 

AEG/AER 

.18 

.857 

1  .746 

NO 

AEG/AEA 

1.12 

.278 

1 .746 

NO 

A  EG  /A  EL 

.43 

.660 

1.771 

NO 

AEW/AER 

-.87 

.410 

-1 .706 

NO 

AEW/AEA 

.58 

.566 

1.706 

NO 

AEW/AEL 

-.56 

.618 

-1 .714 

NO 

AER/AEA 

1 .10 

.288 

1.714 

NO 

AER/AEL 

.34 

.741 

1  .725 

NO 

AEA/AEL 

-.93 

.368 

-1 .725 

NO 
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TABLE  XIV 

Group  Con  romance  to  Pol  icy 


AEG 

AEW 

AER 

AEA 

AH; 

Supervisors 

Program 

Managers 

Number  of 

Subjects 

6 

16 

13 

13 

10 

17 

41 

Exactly 

Conforming 

2 

8 

5 

4 

2 

7 

14 

Conformi ng 

Within  +  .05 

2 

4 

5 

4 

4 

6 

13 

Total  Number 

Conform i ng 

4 

12 

10 

8 

6 

13 

27 

Percent 
Conform! ng 


67 


75  77  62  60 


76 


66 


TABLE  XV 


A  Comparison  of  RM.'ith  Interaction  Terms  and  Without 


No  Policy 


Model 

Overall 

Group 

AEG 

AEW 

AER 

AEA 

ALL 

Supervi sors 

Program 

Managers 

Linear 

.363 

.575 

.416 

.303 

.472 

.509 

.439 

.336 

Inter¬ 

action 

.364 

.578 

.419 

.310 

.491 

.513 

.445 

.337 

Increase 

.001 

.003 

.003 

.007 

.019 

.004 

.006 

.001 

With  1 

3ol icy 

Li  near 

.454 

.536 

.433 

.511 

.606 

.561 

.455 

.675 

Inter- 
acti  on 

.457 

.544 

.439 

.514 

.612 

.566 

.460 

.677 

I ncrease 

.003 

.01? 

.005 

.003 

.006 

.005 

.005 

.002 

In  Chapter  IV,  Summary,  Conclusions,  and  Recommendations,  are  pre¬ 
sented.  The  aim  of  this  chapter  is  to  discuss  how  AE  personnel  rated 
proposals  and  examine  any  unusual  results  which  occurred. 


CHAPTER  IV 


SUMMARY,  CONCLUSIONS, 


RECOMMENDATIONS 


Summary 

This  thesis  was  concerned  with  examining  how  program  managers  and 
supervisors  make  source  selection  decisions.  A  specific  organization 
with  Aeronautical  Systems  Division,  the  Deputy  for  Aeronautical  Equip¬ 
ment  (AE)  was  the  focus  of  this  source  selection  investigation.  Because 
of  time  limitations,  only  AE  could  be  administered  the  exercise. 

Five  evaluation  criteria  were  identified:  technical,  operational, 
logistics,  management,  and  cost.  Two  levels  were  provided  for  each  of 
these  cues:  adequate  or  excellent.  Program  managers  and  supervisors 
were  then  asked  to  rate/evaluate  several  proposals,  one  set.  of  proposals 
when  no  guidance  on  the  relative  importance  of  the  criteria  was  provided 
and  another  set  with  guidance. 

The  results  of  the  exercise  were  critically  examined.  Analysis  and 
results  applicable  to  several  specific  hypotheses  were  presented. 

Consistency  in  rating  proposals  among  individuals  and  among  sub-groups 
was  examined.  Demographic  groups  were  then  compared  for  differences  in 
how  they  rated  proposals.  Predictability  of  proposal  ratings  was  consi¬ 
dered;  i.e.,  strengths  of  specific  evaluation  criteria  in  explaining  the 
decision  model.  Program  managers,  supervisors,  and  three  letter  groups 
were  considered  for  policy  assimilation;  i.e.,  how  well  they  were  able 
to  follow  guidance  on  relative  importance  of  evaluation  criteria.  Last, 
a  non-linear  model  wa s  constructed  and  compared  to  the  standard  linear 
model  which  was  used. 
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Concl usi ons 


The  results  were  extremely  interesting.  In  most  cases  the  hypo¬ 
theses  were  strongly  supported  or  rejected.  As  in  past  research  efforts, 
not  all  of  the  cues  were  necessary  to  adequately  model  a  decision.  Thus, 
people  do  not  always  use  all  the  information  available  to  them  in  making 
a  decision. 

Evaluation  criteria  were  consistently  weighed  by  all  but  one  it. di¬ 
vidual  and  all  but  one  group,  as  indicated  by  R?,s.  The  lower  values 
of  group  R2,s  indicate  that  individuals  have  differing  policies.  The 
exercise  a  1 1  owed  each  individual  maximum  freedom  of  choice  and  anonymity. 

There  were  differences  in  how  three  letter  groups  rated  proposals. 
This  is  not  surprising  because  each  three  letter  group  has  a  different 
function  and  performs  different  types  of  procurement.  AE  is  very  diverse 
and  interests  within  it  vary  widely.  AEG  and  AEA  perform  similar  types 
of  source  selections  as  do  AER,  AEW,  and  AEL.  The  results  showed  that 
AEG  and  AEA  rated  proposals  similarly  but  different  from  the  similar 
rating  policies  of  AER,  AEW,  and  AEL. 

Hypothesis  four  results  are  interesting.  Decisions  were  predictable 
with  three  criteria  when  individuals  were  given  a  policy.  However, 
decisions  were  not  predictable  when  no  policy  wa s  given.  A  possible 
explanation  for  this  is  that  evaluation  criteria  are  always  prioritized 
in  an  RFP.  Perhaps  the  subjects  were  more  comfortable  when  given  a 
pol icy. 

All  groups  examined  significantly  shifted  toward  the  policy.  How¬ 
ever,  there  was  not  a  statistically  significant  difference  in  the  degree 
to  which  they  incorporated  the  policy.  This  seems  unusual.  One  would 
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expect  groups  to  hc.ve  differences  in  ability  to  perform  a  task;  i.e., 
incorporating  a  policy.  Yet  no  significant  differences  wore  uncovered. 

A  possible  explanation  is  that  they  all  did  well  in  accomplishing  a 
not  too  difficult  task. 

The  non -1  inear  model  offered  little  improvement  in  R2.  So,  the 
linear  model  is  an  adequate  instrument  to  model  this  source  selection 
decision.  The  simplicity  of  the  linear  model  is  what  makes  it  so 
appealing.  This  thesis,  as  have  many  others,  shows  the  linear  model 
to  be  quite  strong. 

The  results  showed  that  differences  in  how  three  letter  groups 
within  AE  rate  proposals  exist  even  with  a  prioritized  listing  of  evalua¬ 
tion  criteria.  Management  within  AE  might  desire  to  have  three  letter 
groups  make  very  similar  decisions.  The  fact  that  differences  exist 
implies  that  a  proposal  may  receive  different  ratings  in  different 
three  letter  groups.  This  difference  in  ratings  could  possibly  cause 
problems.  To  combat  this,  Lessons  Learned  briefings  are  conducted  to 
identify  and  document  past  mistakes.  The  hope  is  to  prevent  future 
errors  in  judgment. 

The  result  of  no  difference,  in  the  degree  to  which  supervisors  and 
program  managers  assimilate  a  policy,  is  interesting.  It  is  interesting 
because  the  indication  is  that  supervisors  do  not  do  a  better  job  than 
program  managers  in  assimilating  a  policy.  Supervisors  and  program 
managers  may  be  doing  an  excellent  job  of  communicating  with  each  other. 
Thus,  they  do  not  differ  in  how  they  rate  proposals. 
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Recommendations 


Many  other  organizations  in  the  Air  Force  and  ASD  participate  iri 
source  selection.  It  would  be  interesting  to  examine  more  of  them.  With 
slight  modifications  the  research  exercise  ~n  this  thesis  could  be  used 
to  do  this.  The  exercise  is  simple  and  easy  to  complete. 

In  future  distributions  of  the  exercise,  more  demographic  questions 
should  be  added.  This  would  possibly  enable  explanation  of  things  such 
as  differences  in  three  letter  groups.  Questions  about  how  many  years 
of  experience  and  military  versus  civilian  might  be  added.  An  additional 
question  could  be:  Please  state  the  typical  size  of  source  selections 
you  are  familiar  with. 
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APPEf'DIX  A 


SOURCE  SELECTION  ACTIONS  WITH  AN  EVALUATION  COM!  til  TEE 
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1.  Requirement  Identified 

2.  New  Start  Review 

3.  SSA  and  SS  Chairman  Identified 

4.  Lessons  Learned  Meeting 

5.  Business  Strategy  Meeting 

6.  Initial  Program  Review 

7.  Sources  Sought  -  CBD 

8.  Prepare  Program  Documentation/Pi epare  Executive  Summary  D&F 

9.  SS  Plan  Preparation/Coordination/Approval 

10.  REP  Preparation  (including  model  contract) 

11.  RFP  Coordination/Murder  Board/Approval/Issued 

12.  Selection,  Identification,  Notification  SSEC  Members 

1 3 .  Prepare  Evaluation  Standards/Criteria 

14.  Pre-Evaluation  Meetings  and  Instructions 

15.  Bidders  Briefing 

16.  Letters  of  Intent 

17.  Audit  Request  Prepared 

18.  Pre-Award  Survey  Request  Prepared 

19.  EEO  Request  Prepared 

20.  Pricing  Coordination 

21.  Administration  Established 

22.  SS  Facilities  Planned 

23.  Receive/Screen  Proposals 

24.  Evaluations 

25.  Audit/PAS/EEO  Requests  Sent 
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26.  Cl /DR/MR  Transmit  till  Flow 

27.  Preparation  of  evaluation  Report 

28.  Tact  Finding  Discussions  (Technical) 

29.  Revise  Model  Contracts 

30.  Preliminary  Briefing  to  SSA 

31 .  Cl /DR/MR  Flow  Complete 

32.  Technical  Evaluation  Complete 

33.  Negotiation  of  Technical ,  Management,  TOC 

34.  Cost  Discussions 

35.  Model  Contracts  Completed 

36.  Prel iminary  Review 

37.  BAFO  Request  Issued 

38.  BAFO  Received,  Analyzed/ Summarized 

39.  Summary  Report  Prepared 

40.  PAR  Prepared 

41.  Final  Review 

42.  Final  Briefing  to  SSA 

43.  Decision  Documents 

44.  Loser  Letters/Contract  Award 

45.  Debriefings 

46.  Lessons  Learned 
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APPENDIX  C 


SOURCE  .ELECTION  EXERCI 


a  scu;iCi;  selection  excgoiec  foe  ::wg.\a:  j  .ai:ag:,rs 
IK  THli  DEPUTY  FOR  AERONAUTICAL  E^Ui  j ...ENT 


This  1:;  EOT  A  Qiiiv a  i  yL.  It  is  a  source  selection 
exercise  to  investigate  how  proj  r<  .  ..  u  rs  utilize  eval¬ 
uation  criteria  in  evaluatii  proposals.  The  d  t«  collected 
will  support  a  rese  rch  ei  ort  sanctioned  by  AS  and  a  master's 
thesis  at  the  Air  Force  Insti  tute  of  Technology  but  you  w?  11 
not  be  identified  in  the  f 3  ;  ‘  r  rt.  four  cooperation  is, 
there  ’ore,  sinc<  rely  r<  -\  .  Youi  input  to  this  research 

will  be  kept  strictly  ccgvcj  ’"*• 

The  dollar  amount  of  each  proper;  -.1  to  bo  evaluated  in 
this  exercise  is  $400,000,  Thus,  nor  •  of  th *  formal  co 
mittees  (Source  Selection  Advisory  Council,  Source  Selection 
Evaluation  -card,  and  Source  Selection  Evaluation  Com!.',  it  tee ) 
exist  for  this  exercise.  The  Source  Selection  Authority 
rests  with  the  Irocuri;  -  Contracting  Officer, 

The  exorcise  is  divided  into  three  sections.  In  Section 
you  are  asked  to  identify  your  organ.*!  cation.  Section  li  in¬ 
volves  evaluating  proposals  with  respect  to  several  evaluation 
criteria.  The  evaluation  criteria  are  intentionally  -not  prior 
itized  in  this  section.  In  Section  III  a  policy  is  stated 
in  relation  to  the  evaluation  criteria.  You  arc  then  asked 
to  evaluate  additional  proposals.  The  stated  policy  should 
be  considered  when  evaluating  proposals, 

There  are  no  "correct"  or  "incorrect”  answers  so  please 
respond  as  candidly  as  possible.  The  information  provided 
by  you  and  other  respondents  will  be  combined  to  statistically 
test  hypotheses  about  how  information  is  used  by  i.ro' ra.n.  man¬ 
agers  in  AE  to  evaluate  offerors'  proposals,  Each  individ¬ 
ual's  exercise  will  be  destroyed  once  the  responses  have  been 
keypunched  onto  computer  cards.  This  is  your  opportunity  to 
contribute  to  an  experimental  analysis  of  p  rt  of  the  source 
selection  process. 


ALL  DECISIONS  RELATE  TO  EVALUATING  PROPOSALS 


i lease  return  this  exercise  within  ten  days  so  that  it 
may  be  included  in  the  research,  kritc  any  comments  you  nay 
have  on  the  last  page.  Thank  you. 
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PRIVACY  STATES!  NT 


3n  accordance  with  paragraph  30,  APR  12-35<  tJie  fol¬ 
lowing  information  is  provided  as  required  by  the  Privacy 
Act  Of  1974 ; 

a.  Authority 

(1) 4  U.S.C.  301*  Departmental  Re  aula  IF  one  t  and/or 

(2)  10  U.3.C.  80-12,  SecTvjtrrx  of  the  Air  Force , 
Powers  rod  Duties  ,  De.l  opati  or.  bv« 

b.  Principal  purposes.  The  survey  is  being  conducted 
to  collect  information  to  be  used  in  research  aimed  at,  il¬ 
luminating  and  providing  inputs  to  the  solution  of  problems 
of  interest  to  the  Air  Force  and/or  DOD. 

c.  Routine  Uses.  The  survey  data  will  be  converted  to 
information  for  use  in  research  of  management  related  prob¬ 
lems,  Results  of  the  research  based  on  the  data  provided, 
will  be  included  in  written  Master's  thesis  and  may  also  be 
included  in  published  articles,  reports,  or  texts.  Distri¬ 
bution  of  the  results  of  the  research,  based  on  the  survey 
data,  whether  in  written  form  or  orally  presented,  will  be 
unlimited. 

d.  Participation  in  this  survey  is  entirely  voluntary. 

e.  Ko  adverse  action  of  any  kind  may  be  taken  against 
an  individual  who  elects  to  participate  in  any  or  all  of 
this  survey. 
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II. 


•oi KC3  OKI  ,;:c?ro.N  ikjtru'jtj o.\b 


This  section  consists  of  a  source  selection  exercise. 
During  the  exercise,  you  should  assure  that  you  are  respon¬ 
sible  for  evaluating  offerors'  proposals.  The  only  real 
differences  in  these  proposals  are  the  degree  to  which  each 
individual  evaluation  criteria  is  net  or  exceeded.  Assume 
the  proposals  do  not  differ  on  other  attributes. 

These  five  evaluation  criteria  are  described  as  ADEQUATE 
ox'  EXC  .T.  Bach  proposal  has  been  rated  in  relation  to 
these  criteria,  for  this  exercise 
I... ALL-'  ACCEPT  A. bJ  E  VALUE  OR  hb QUIRK 


ADEQUATE  means 

,  .  >  I  •  Hi  A  \J  *->  1-  •  1.  L  j  . 


"  C  0  N  5 1 DE  R  A  ELY 
QUIREMENT.  " 


BETTER  THAN  vll-  If  ALLY  ACCEPTABLE  VALUE 


etc 

eans 
OR  RE- 


An  explanation  of  the  evaluation  criteria  is  presented 
below.  They  apply  to  all  dec  it 
between  adequate  and  excellent 


refers  to  the  s 


ion 

e .  ]  t 

oi .oral::  w 

hich 

fall 

are 

not  a 

part  of 

this 

exen 

<?  s 

oundiv 

sr ,  techr 

3  c  -  3 

:  £  r  3 

i  c 

desigi 

c)  oi  (  n . 

It 

is  me 

—  To  elm  io  a  1 .  Ti  \ ' 

and  level  of  risl 
sured  as  the  degree  to  which  technical  qualities  mee 
ceed  specifications  in  all  technics]  areas. 


or 


- -Orera  ti  ona  1 .  This 
of  the  proposed  svr-r 


rofers  to  the  functional  jsrfori-o 
;rn  to  meet  the  requirements  of  t  le 


m 


itary  mission.  It  is  measured  as  the  degree  to  which  esti¬ 
mated  performance  meets  or  exceeds  all  performance  specifi¬ 
cations. 


—  ,  r>rr 


3  sties.  This  refers  to  the  supported j lity  of  the  pro¬ 


posed  system.  It  is  reflected  in  the  extent  of  attention  to 
the  reliability /maintainability  characteristics  of  design. 


It  is  r 
repair 
dards . 


ensured 


the  degree  to  which  estimated  r... 


;n  t  in¬ 


to 


md  mean  tii.e  between  failures  meets  or  exceeds  cPri- 


-- vane re; -ent.  This  refers  to  the  qualifications  of  the  of¬ 
ferors'  management  personnel  and  the  evidence  of  adequate 
planning  and  control  efforts.  It  is  measured  by  the  extent 
of  evidence  in  the  proposal  of  good  management  planning  and 
control  systems  and  by  the  offerors'  past  record  of  contract 
performance , 

--Cost.  This  refers  to  the  current  best  estimates  of  life 
cycle  cost  and  the  soundness  of  the  contract  cost  estimates. 
It  is  measured  as  the  degree  to  which  sound  estimates  arc 
within  cost  targets. 
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T,'or  each  proposal,  you  will  bo  provided  ratings  in  re- 
lntion  to  each  of  tile  five  evaluation  criteria,  leu  will 
then  bo  asked  to  evaluate  the  proposal.  Indicate  your 
evaluation  of  each  proposal  on  the  scale  below  it,  A  sample 
scale  is  presented  below, 

012345678910 


Highly 

Undesirable 


Highly 

Desirable 


You  should  circle  the  number  that  best  indicates  your 
evaluation  of  each  proposal,  Make  each  decision  one  at  a 
time  and  independently  of  the  others.  Do  not  change  a  de¬ 
cision  once  you  have  made  it.  Work  at  a  brisk  pace,  but 
don’t  hurry  your  decisions.  Complete  EVERY  case  as  each 
case  is  DIFFERENT.  The  proposals  begin  on  tl  •  1  'oil or.  ing 
page . 

Notes  The  evaluation  criteria  are  intc  .. •  one  1 1;/  not 
prioritized  in  this  section.  Order  of  listii  v  s  no  sig¬ 
nificance  . 
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1  EG;.  ‘.I  1 


Technical  . ADEQUATE 

Operational  . .  ADEN}  DATE 

Logistics  . . ADEQUATE 

Management  . . EXCELLENT 

Cotit  . . ADEQUATE 

Indicate  below  your  overall  evaluation  of  this  proposal. 

0123456789  10 

Highly  Highly 

Undesirable  Desirable 

PR  OPOSAL  7';  7, 

Technical  . EXCELLENT 

Operational  . . ADEQUATE 

Logistics  . ADEQUATE 

Management  . EXCELLENT 

Cost  . . EXCELLENT 

Indicate  below  your  overall  evaluation  of  this  proposal. 

0  1  2  3  4  5  6  7  8  9  10 

Highly  Highly 

Undesirable  Desirable 


PROPOSAL  #3 

Technical  . . . EXCELLENT 

Operational  . . EXCELLENT 

Logistics  . EXCELLENT 

Management . EXCELLENT 

Cost . EXCELLENT 

Indicate  below  your  overall  evaluation  of  this  proposal. 

0123456789  10 

Highly  Highly 

Undesirable  Desirable 

PROPOSAL  #4 

Techn.i cal  . . ADEQUATE 

Opera  tional  . EXCELLENT 

Logistics  .  ADEQUATE 

Management . . . ADEQUATE 

Cost  .  ADEQUATE 

Indicate  below  your  overall  evaluation  of  this  pi’oposal, 

0123^56789  10 

Highly  Highly 

Undesirable  Desirable 
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PROPOSAL  T'5 


Technical  . . . . .  ADEQUA'J n 

Operational  . . . . . . .  AD  iQUA'is 

Logistics  . EXCELLENT 

Management  . . EXCELLENT 

Cost  . ADEQUATE 

Indicate  below  your  overall  evaluation  of  this  proposal. 

012345678910 

Highly  Highly 

Undesirable  Desirable 


PROPOSAL  #6 

Technical  ...............  . . ADEQUATE 

Operational  . . .  .  EXCELLED T 

Logistics  . . . .  . . .  EXCELLED T 

Management  . . ADEQUATE 

Cost  . ADEQUATE 

Indicate  below  your  overall  evaluation  of  this  proposal. 

01  23^56  7  8910 

Highly  '  Highly 

Undesirable  Desirable 


PROPOSAL  #7 

Technical  . . ADEQUATE 

Operational  . . .  .  EXCELLENT 

Logistics  . . . .  EXCELLED T 

1  ’  a  n  s  go  .[ieno  ..............................  aC  f .  1 1  ■  .  < ,  I 

Cost . . . ADEQUATE 

Indicate  below  your  overall  evaluation  of  this  proposal. 

01  2  3  4  567  8  9  10 

Highly  Highly 

Undesirable  Desirable 


PROPOSAL  #8 

Technical  ,  . . EXCELLENT 

Operational  . ADEQUATE 

Logistics  . ' .  EXCELLENT 

Management  . EXCELLENT 

Cost  . EXCELLENT 

Indicate  below  your  overall  evaluation  of  this  proposal . 

01  23456789  10 

Highly  Highly 

Undesirable  Desirable 


PROPOSAL  ti  9 


Technic  al  . .  EXCKI/LiaT 

Operational  . .  EXCELLENT 

Logistics  . . .  EXCELL'L.T 

!»io  lie*  \ 11 1  ..............................  .  A  T  !'-j  QUll  i  i-V 

.'Oct  ............  xvy.oEEJ.Ei  v  x 

Indicate  below  your  overall  evaluation  of  this  proposal , 

0123  4  56?8910 

Highly  Highly 

Undesirable  Desir;  t 


PR  0  x  U  »i>  A  L  // 1 0 

Technical  . . . . 

Operational  . . . 

Logistics  .  . . . 

Management  .  . ................  . 

Cost  . . . . . 

Indicate  below  your  overall  evaluation  of  this  proposal. 


0123  4  5678910 

Highly  Highly 

Unde  s irable  De s irao 

PROPOSAL  r  1 1 

Technical  . . EXCELLENT 

Operational  . . EXCELLENT 

Logistics  . . ADEQUATE 

Management  . EXCELLENT 

Cost . EXCELLENT 

Indicate  below  your  overall  evaluation  of  this  proposal. 

0123^5678910 

Highly  High] 

Undesirable  Desire 


PROPOSAL  112 

Technical  . . EXCELLENT 

Operational  . . ADEQUATE 

Logistics  . ADEQUATE 

Management . ADEQUATE 

Cost . . . EXCELLENT 

Indicate  below  your  overall  evaluation  of  this  proposal. 

0123456789  10 

Highly  Hirhl 

Undesirable  Desir 
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EXCELLENT 
,  ADEQ1  k'l 
EXCEL!  S*.  x 
■  ADEQUATE 
EXCELLENT 


PR  01  DEAL  .  13 


Te c hn ical  . . .  ADK Q UA 92 

Operational  . . . .  EXCSLLSS  f 

Logistics  .  . . . . . .  ADEQUATE 

Management  . . . .  EACELLEY.T 

Cost  . .  . . . . .  .  ADEQUATE 

Indicate  below  your  overall  evaluation  of  this  proposal. 

0123^56789  10 

Highly  Highly 

Undesirable  Des liable 


PROPOSAL  pj\ 

Technical  . . ADEQUATE 

Operational  . . . .  ADEQUA  AS 

Logistics  . EXCELLENT 

Management  . . ADEQUATE 

Cost  . . ADEQUATE 

Indicate  below  your  overall  evaluation  of  this  proposal, 

012345678910 

Highly  Highly 

Undesirable  Desirable 


PROPOSAL  pi 5 

Technical . . . . .  ADEQUATE 

Operational  . . . . .  *  ADEQUATE 

Logistics  . . . .  ADEQUATE 

Management  . . .  ADEQ17  TE 

Cost  . . .  ADEQUATE 

Indicate  below  your  overall  evaluation  of  this  proposal. 

0123456789  10 

Highly  Highly 

Undesirable  Desirable 


PROPOSAL  pi  6 


Technical  . EXCEI.lEhT 

Operational  . . . . >  •  EXCELlLLf 

Logistics . . . .  ADEQUA  To 

Management  . . ADEQUATE 

Cost' . EXCELLENT 

Indicate  below  your  overall  evaluation  of  this  proposal. 

012345678910 


Highly 

Undesirable 


Highly 

Desirable 
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PRO!  0  :  .  'i  7 


IROK'JAJ 


r 


PROPOSAL  /  21 

Technical  .  ADS  S'  A'. 

Operational  . .  ...........  . . .  KXCHl  ! 

Login ti  cs  . . . .  EXCEL;. 

Manage  "nent . . . .  EXCELL*  .  i1 

Cost.  ...................  .  ...  . .  J  jXO— .LEs  1\’X 

Indicate  below  your  overall  evaluation  of  this  proposal. 


2  3 


Highly 
Uncles  irabl  e 


Highly 

Desiral 


PROPOSAL  /;  22 

Technical  . . . . .  EXOE'LL"  I.T 

Operational  . . . .  EXCEL.! 

Logistics  . . . .  EXCELL..;.  p 

Management . . . . .  ADEQUATE 

Cost  . . .  ADEQUATE 

Indicate  below  your  overall  evaluation  of  this  proposal, 

oi  2345678910 


Highly 
Undos irable 


Highly 
Do  sir  ... 


PROPOSAL  :  2 3 


Technical  . . . . . .  EXCEL!  A.! 

Operational  . EXCELLENT 

Logistics  . EXCELLE..T 

Management  . . . . .  EXOELLE  -  T 

Cost  ADEQUATE 

Indicate  below  your  overall  evaluation  cf  this  proposal 


2  3 


7  8 


Highly 

Undesirable 


n  :i.°  i 
Do  r.  5  r  .  1 


PROruSAL  ...  24 


Technical  . ADEQUATE 

Operational  . . . .  EXCEL;:... f 

Logistics  . .  EXC:  .oj.EM’ 

Management  . . . .  .  ADEQUATE 

Cost . EXCELLENT 

Indicate  below  your  overall  evaluation  of  this  proposal. 


Hi  hi; 


Highly 

Undesirable 
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pr  oi  cial  25 

Technic;  .1.  . . . . .  APE QP ATE 

Operational  . . . . .  EXCEL.'. -X;  'T 

Logistics  . .  ADEQUATE 

Kanngement  . . ADEQUATE 

Cost . EXCELLED! 

Indicate  below  your  overall  evaluation  of  this  proposal, 

0  1  2345678910 

Highly  Highly 

Unde  s  i.ra  ble  L)e  s  irable 


PROPOSAL  ii 26 

Technical  . . ADEQUATE 

Operational  . . ADEQUATE 

Log  3  sties  . ADEQUATE 

Management  . . ADEQUATE 

Cost  . . EXCELLENT 

Indicate  below  your  overall  evaluation  of  this  proposal. 

01  2  34  5  6  7  8  9  10 

Highly  Highly 

Undesirable  Desirable 


PROPOSAL  /X27 

Technical  . EXCELLEKT 

Operational  . EXCELLENT 

Logistics  . . ADEQUATE 

Management . . . EXCELLED! 

Cost ' . ADEQUATE 

Indicate  be low  your  overall  evaluation  of  this  proposal. 

01  23  4  5678910 

Highly  Highly 

Undesirable  Desirable 


PR0P03AL  28 

Technical  ................................  ADEQUATE 

Operational  . EXCELLEKT 

Logistics  . ADEQUATE 

Management  . . . . .  .  EXCELLED! 

Cost  EXCnLLElvT 

Indicate  below  your  overall  evaluation  of  this  proposal, 

0  1  2  3  4  5  6  7  8  9  10 

Highly  Highly 

Undes irable  Dcs i rable 
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}  ilOPCDAL  .  29 


Technics]  . . 

}<■  /f]  j  j  • 

Oyer  •lien:  .1 
Logistics  , , 

.....  ADEQUATE 

Management  . 

....  EXCEL!  ■(  'AT 

Cost  . 

v  ♦  *  »  »  A D QUA’.-  ; 

Indicate  below 

your  overall  evaluation 

of  thin-  proposal. 

0  12 

3  4  5  6? 

8  9  10 

Highly 

Highly 

Undesirable 

Desirable 

I  i'v  0 1  ‘  0  A  X ;  ]\  j  0 

Technical  .  ....  .........  . . ......  ADEQUATE 

Operational  . . .  ADEQUATE 


Management  , , 

.  .  EXCELLENT 

Cost 

.  .  EXCELLENT 

Indicate  below 

your  overall  evaluation 

of 

this  proposal. 

0  1  2 

3  4  5 

6  7 

8  9  10 

H iph ly 

Highly 

U.ndes  irable 

Desirable 

IRQ  LOCAL  ,31 

Technical  , . 

.  .  EXCELLENT 

Operational 

*  t 

.  .  .  ADEQUATE 

logistics  . . 

,  ,  ,  ADA Oi  i  ■'  ■  i 1  j*l 

[•an  age  rent  . 

.  .  .  A DHQ’.i  •’.  i‘:  . 

Cost . . 

,  .  .  ADEQUATE 

Indicate  be law 

your  overul  .1  eva ]  ua  t.ion 

of 

this  proposal. 

0  12 

3  4  5 

6  7 

8910 

Highly 

Highly 

Undesirable 

Desirable 

I  R0103AL  ...  32 

/ 

Technical  . . 

0  4 

.  .  .  ADEQUATE 

Operation? 1 

,  ADEQUATE 

Logistics  ,  , 

,  .  EXCEL IjAKT 

Management  , 

t  4 

,  .  .  ADEQUATE 

Cost  . 

c  ,  EXCELLED  C 

Indicate  below 

your  overall  eva 

luati on 

of 

this  proposal. 

0  1  2 

3  4  5 

6  ? 

8  9  10 

Highly 
Unde  drab]  o 


Highly 

Desirable 


III.  I  v  LICf  -STA'i  !•  wnT 

The  evaluation  criteria  arc  lie ted  in  decreasing  order 
of  importance.  This  priority  applies  when  evaluating  all 
proposals  in  this  section. 

J .  Operational 
?.  Technical 

3.  Cost 

4.  Logistics 

5.  Management 


All  instructions 


in  Section  II  also  apply  to  this  section, 


except  in  relation  to 


priority  of  evaluation  criteria. 


how¬ 


ever,  DO  NOT  refer  to  the  proposals  in 
uating  the  proposals  in  this  section, 
on  the  following  page. 


Section  II  when  cval- 
The  proposals  begin 
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Technical  . .  ADEQUATE 

0}  .'C  X  l  i . .  .  0)  1  il  1  ....tti.it.ii.i..............  -  *  - 

Logistics  . . . .  EXCELLEKT 

Mans  g;e  non  t  . . .  EaC EL  LET 

Cost  . .  AD;  'QUA'i E 

Indicate  below  your  overall  evaluation  of  this  proposal. 

0123^5678910 

Hipli.lv  Highly 

Undesirable  Desirable 


PROPOSAL  a  2 

Technical  . . .  •  •  ADEQUATE 

Operational  EXC EL  .  ■ .  .T 

Logistics  .  . . . . .  •  •  ADLQu.'  TE 

Management  . .  EXCEL1E  ■  T 

Cost' . . . . .  ADEQUATE 

Indicate  below  your  overall  evaluation  of  this  proposal. 

0123  456789  10 

Highly  Highly 

Unde  s irable  be s irable 


PROPOSAL  £3 

Technical . . . ADEQUATE 

Opera  wioncil  AD.-vOe/n  -« 

Logistics.  .......  .......................  EXCEi rEI.T 

Management  . ADEQUATE 

Cost  . ADEQUATE 

Indicate  below  your  overall  evaluation  of  this  proposal. 

0123456789  10 


Highly 
Undesirable 

lWro3Tf?4 


Technical . . . EXCELLEKT 

Operational . . . KXCEL.iEKT 

Logistics  . . . *  EXCEL!  a.ET 

Management  . ADEQUATE 

Cost  EXCEon^NT 

Indicate  below  your  overall  evaluation  of  this  proposal. 

1  2  3  4  5678910 


Highly 

Desirable 
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Highly 

Desirable 


1  .-;0i  V-/.  *  1 1 


5 


Technical  . EXCSLLEKT 

Operational  . , .  ADEQUATE 

Logistics  .  EXCELLED'! 

Management  . ADEQUATE 

Cost  . EXCELLED! 

Indicate  below  your  overall  evaluation  of  this  proposal. 

oi  2345678910 


Highly 

Undesirable 


Highly 

Desirable 


PROPOSAL  #6 


Technical  . . EXCELLED! 

Operational  . . EXCELLENT 

Logistics  . . ADEQUATE 

Management  . . ADEQUATE 

Cost  . . . . . .  .  ,  ,  .  EXCELLED! 

Indicate  below  your  overall  evaluation  of  this  proposal, 

0123  4  5678910 

Highly  Highly 

Undesirable  Desirable 


PROPOSAL  r‘7 

Technical  . . ADEQUATE 

Operational  . .  ADEQUATE 

Logistics  . ADEQUATE 

Management  . . EXCELLENT 

Cost  . ADEQUATE 

Indicate  below  your  overall  evaluation  of  this  proposal, 

01  2345678910 

Highly  Highly 

Undesirable  Desirable 


PROPOSAL  g 8 

Technical  . . ADEQUATE 

Operational  . ADEQUATE 

Logis  Lies  . . . .  .  ADEQUATE 

Management  . ADEQUATE 

Cost  . ADEQUATE 

Indicate  below  your  overall  evaluation  of  this  proposal. 

012345678910 

Highly  Highly 

Undesirable  Desirable 
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PROPOSAL  x9 


Technical  . 
Operational 

LogXS  t  J.  o  3  ■ 

Management 

C  0  S  t  ...... 


Ai^QUAjJ. 

j  ■  i  »  C 

KXCi  ,'LLX:  •  T 
ADEQUATE 

/.DR!''  A .  ■ 


Indicate  below  your 
0  12  3 


overall  evaluation  of  this  p  op  :,al. 

4  5  6  ?  6  9  10 


Highly 

Undesirable 


Highly 

Desirable 


1  ..01  0  . 


Technical  . . . . . .  .  . . . . . .•«••• 

Operational 

Logistics  ........ t ..... c .  - .  r « 

Management  . . .  .  .  . 

Cost  ...........  ...... ....... ......... 


this  proposal. 

8  9  10 

Highly 

Desirable 


PROPOSAL  if l  i 


Indicate  below  your  overall  evaluation  of 

0  1  2  3  4  5  6  7 

Highly 

Undesirable 


«  AD  A.  "..AX 

.  ADEQUATE 
EXCE  LLXNT 
EXCELLENT 


Technical  .... 
Operational  , . 
Logistics  , . . . 
Management  . . . 

Cost  ......... 


a  e  ccacraatte* 


act  ttaata**** 
aaaoctatettaa 


f  ♦  t  »  f  p  e  ••  •  t  »  #  ♦  c 


*  c  e  c  t  *  tecrace 


ADEQUATE 
ADEQUATE 
EXCEL!  -EKT 
EXCEL!.  AWT 
ADEQUATE 


Indicate  below  your  overall  eve 


li¬ 


lt  ion  of  this  proposal 


0 


3  4  5678 


9  1 0 


Highly 

Undesirable 


Highly 

Desirable 


PROLOG AL  l!  1 2 


Technical  . 
Operational 
Logistics  . 
Management 

Cost . . 


EXCELLENT 
EXCELLED’ T 
AI  SQUATE 
E  XOEL  L1  j  1  >  T 
EXCELLENT 


Indicate  below  your  overall  evaluation  of  this  proposal. 

0  12  3  '(•  5  6  7  8  9  10 


Highly 

Undesirable 
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Highly 

Desirable 


I ROPOSAL  i:  1 3 


Technical  . . . . . . .  ADEQUATE 

Operational  . . . .  EXCEEiEL  i' 

Logistics  . ADEQUATE 

Management . . . . .  A DEQUA  TH 

Cost  . . . .  ALL QUA:  E 

Indicate  below  your  overall  evaluation  of  this  proposal. 

0  1  2  3  4  5  6  ?8910 

Highly  Highly 

Undesirable  Desirable 


PROPOSAL  flh 

Technical . . . . . . .  EX  OH. ILL’ AT 

Operational  . ADEQUATE 

Logistics  . . EXCELLENT 

Management  . . . . . . . .  .  EXCELLENT 

Cost  . EXCELLENT 

Indicate  below  your  overall  evaluation  of  this  proposal. 

0123456  7  8910 

Highly  Highly 

Undesirable  Desirable 


*'  *  '  ‘ . PROPOSAL  #15 

Technical  . . .  .  EXCELLENT 

Operational  . . .  ADS QUA i  i 

Logistics  . ADEQUATE 

Management  . . .  ADD  QUA  IE 

Cost  .  . . EXCELLENT 

Indicate  below  your  overall  evaluation  of  this  proposal. 

0  1  2  3  4  5  6  7  8  9  10 

Highly  Highly 

Undesirable  Desirable 


PROPOSAL  #16 

Technical  . . EXCELLENT 

Operational  . . EXCELLENT 

Logistics  . . EXCELLENT 

Management  . .  EXCELLENT 

Cost  . EXCELLENT 

Indicate  below  your  overall  evaluation  of  this  proposal. 

01  2345678910 

Highly  Highly 

Undesirable  Desirable 
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j  i  ■.  01  Coal  1 7 


Technical  ................... 

k  '/c  ■  '  i  ‘i  y 

Operational  . . . 

Logistics  . . . 

Vanareiiient  . . 

Cost . . . 

. ,  /.  Df’gA  i  TV 

Indicate  below  your  overall  eva 

.1  eat  ion 

of  this  proposal . 

0  1  2345 

6  7 

8  9  10 

Highly 

Undos irable 

Highly 

Desirable 

VR OrOSAL  //  \ 3 

Technical  . . 

.  .  .  .  .  li/CKL'  H 

Operational  . . . . 

......  A  H  .  0 1  ]  :\  « * 

Logistics  ................... 

Management  . . 

A 1  '  0  \  I A  m  •  > 

Cost . . . 

1  r  t  •  ♦  »  ^  U  .'i,  jL .  j 

Indicate  below  your  overall  eva 

luation 

of  this  proposal. 

0  12  3  4  5 

6  7 

8  9  10 

Highly 

Undesirable 

Highly 

Desirable 

PROPOSAL  19 

Technical  . . . . . 

.  ....  EXCELLENT 

Operational  . . . 

.  t  t  .  .  A  "n^QUAVK 

Lo°'  i  s  t  i  c.  3 

ban age me nt . . . 

.  #  ,  / , XC  •  i  '■  )  .  . 

Cost  . 

.  .  .  .  .  A  DK  QUA  'Y;\ 

Indicate  below  your  overall  evaluation 

of  this  proposal. 

0  12  3  4  5 

6  7 

8  9  10 

Highly 

Undesirable 

Highly 

Desirable 

PROPOSAL  ,v  20 

Technical  . . . . 

, , , . ,  ADEQUATE 

Operational  ................. 

....  EXCEPT  Ei  .'P 

Lorristics  . . . 

.....  ADI*)  QUA  Tf£ 

Management . . 

....  EXCELLENT 

Cost  . 

Indicate  below  your  overall  evaluation 

of  this  proposal. 

0  1  2  3  4  5 

6  7 

8910 

Highly 

Desirable 


Mi ghly 
Urdes  irable 
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Indicate  below  your  overall  evaluation  of  thin  proposal, 

01  2345678910 


01  2345678910 

Highly  Highly 

Unde  s  irable  Des  ira'ole 

'~W6f6jnrj.l£: 

Technical . .  EXCELLENT 

Operational  . . . . . , .  .  EXCELLENT 

Logistics  . EXCELLENT 

Management  . . .  .  . . .  EXCELLENT 

Cost" . ADEQUATE 

Indicate  below  your  overall  evaluation  of  this  proposal. 

0123  4  56789  10 

High!'-  Highly 

Undesirable-  Desirable 

. 

Technical  . . . . .  ADEQUATE 

Operational  . . ADEQUATE 

Logistics  . EXCELLENT 

Management  . .  ADEQUATE 

Cost  EXCELLENT 

Indicate  below  your  overall  evaluation  of  this  proposal. 

0  1  2  3  4  5  6  7  8  9  10 

Highly  Highly 

Undesirable  Desirable 

PHOTOS AL  #24 

Technical  . ADEQUATE 

Operational  . . EXCELLENT 

Logistics  . . EXCELLENT 

Management.  . . EXCELLENT 

Cost  , EXOxiLLnnT 

Indicate  below  your  overall  evaluation  of  this  proposal. 

012345678910 

Highly  Highly 

Undesirable  Desirable 
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IRGIOSAL  /  25 


Technical  . ADEQUATE 

Operational  . . . . .  ADE'QUA'iE 

Logistics  . . .  AD SQUAT  2 

Management  . .  ADEQUATE 

Coo  t  E/OE  LLL..T 

Indicate  below  your  overall  evaluation  of  this  proposal. 

0123456  7  8910 

Highly  Highly 

Undesirable  Desirable 


’PROPOSAL  ];  2*6 


Technical  . ADEQUATE* 

Operational  . . ADEQUATE 

Logistics  . ADEQUATE 

Management  . . . .  .  EXCELLED T 

Cost  . EXCELLENT 

Indicate  below  your  overall  evaluation  of  this  proposal. 

0123  4  5678910 

Highly  Highly 

Undesirable  Desirable 


PROPOSAL  i;  27 


Technical  ,  . . EXCELLED! 

Operational  . . ADEQUATE 

Logistics  . . ADEQUATE 

Management  . EXCELLENT 

Cost  . ADEQUATE 

Indicate  below  your  overall  evaluation  of  this  proposal. 

0123^56789  10 

Highly  Highly 

Undesirable  Desirable 


PROPOSAL  ,728 


Technical  ADEQUATE 

Operati onal  .  . . ADEQUATE 

Logistics  . EXCELLENT 

Management  . EXCELLENT 

Cost  ........................... .........  EXCELLENT 

Indicate  below  your  overall  evaluation  of  this  proposal. 

012345678910 

Highly  Highly 

Undesirable  Dos  .irabl  e 
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JR01  Cvj.'J 


Technical  ...... . . . .  . .  ADEQUATE 

Operation-’ 1  . ........ . .  EXCELLENT 

Logistics  . . . . . . .  ADEQUATE 

Management  . . . . . . . . .  ADEQUAT., 

Coot  E XC E  1  jia . . T 

Indicate  below  your  overall  evaluation  of  this  proposal . 

0123456789  10 


Highly 
Undos I  r able 


Highly 

Desirable 


FROI'OoAL  it  30 

Technical  .  EXCELLENT 

Operational  . . . . . .  . . EXCELLED T 

Logistics  . ADEQUATE 

Management  . ADEQUATE 

Cost  . ADEQUATE 

Indicate  below  your  overall  evaluation  of  this  proposal . 


0j234 


8  9  10 


Highly 

Undesirable 


Highly 

Desirable 


PR010SAL  y  31 


Technical  . , 
Operat ional 
Logistics  .. 


EXCELLENT 

V  VPTT  1  n 

ji-iL-t  1  .  i. 


Management  . . .  ,  ,  ,  . . .  ADEQUATE 

^  Cost  . . . .  ADEQUATE 

\ 

Indicate  below  your  overall  cv  ,"  nation  of  this  proposal . 


H ighly 
Undes i  ratio 


Highly 
Do  sir;;  hie 


J'ROJ  03AL  --3: 


Techn.i  cal . . . . 

Operational  . . . . 

Logistics  . . .  ,  ,  . . .  . 

manage  merit  ,  . . ,,,,,,,,,, . 


I  DEC UAT2 


yrp  ■ 


:  ,v X i' 


jo.'i J.v.L i  XT 


Indicate  be  lev.’  your  overall  evaluation  of  this  proposal. 

0123450789  10 


Highly 
Unde  s  i  ry  1>  1.* 


jug',  ly 
Dec  ire bln 


With  No  S tat e d  Pol i cy 


Proposal 

Number 

Rati  i 

rj' 

•> 

Orthogonal 

Vectors 

1 

A 

A 

A 

E 

A 

0 

0 

0 

1 

0 

2 

E 

A 

A 

E 

E 

1 

0 

0 

1 

1 

3 

E 

E 

E 

E 

E 

1 

1 

1 

1 

1 

4 

A 

E 

A 

A 

A 

0 

1 

0 

0 

0 

5 

A 

A 

E 

r 

L 

A 

0 

0 

1 

1 

0 

6 

A 

E 

E 

A 

A 

0 

1 

1 

0 

0 

7 

A 

E 

E 

E 

A 

0 

1 

1 

1 

0 

8 

E 

A 

E 

E 

E 

1 

0 

1 

1 

1 

9 

E 

E 

E 

A 

E 

1 

1 

1 

0 

1 

10 

E 

A 

E 

A 

E 

1 

0 

1 

0 

1 

11 

E 

E 

A 

E 

E 

1 

1 

0 

1 

1 

12 

E 

A 

A 

A 

E 

1 

0 

0 

0 

1 

13 

A 

E 

A 

E 

A 

0 

1 

0 

1 

0 

14 

A 

A 

E 

A 

A 

0 

0 

1 

0 

0 

15 

A 

A 

A 

A 

A 

0 

0 

0 

0 

0 

16 

E 

E 

A 

A 

E 

1 

1 

0 

0 

1 

17 

A 

A 

E 

E 

E 

0 

0 

1 

1 

1 

18 

E 

A 

E 

E 

A 

1 

0 

1 

1 

0 

19 

E 

A 

F 

A 

A 

1 

0 

1 

0 

0 

20 

E 

E 

A 

A 

A 

1 

1 

0 

0 

0 

21 

A 

E 

E 

E 

E 

0 

1 

1 

1 

1 

22 

E 

E 

E 

A 

A 

1 

1 

1 

0 

0 

23 

E 

E 

E 

E 

A 

1 

1 

1 

1 

0 

24 

A 

F. 

E 

A 

L 

0 

1 

1 

0 

1 

25 

A 

E 

A 

A 

E 

0 

1 

0 

0 

1 

26 

A 

A 

A 

A 

E 

0 

0 

0 

0 

1 

27 

E 

E 

A 

E 

A 

1 

1 

0 

1 

0 

28 

A 

E 

A 

E 

E 

0 

1 

0 

1 

1 

29 

E 

A 

A 

E 

A 

1 

0 

0 

1 

0 

30 

A 

A 

A 

E 

E 

0 

0 

0 

1 

1 

31 

E 

A 

A 

A 

A 

1 

0 

0 

0 

0 

32 

A 

A 

E 

A 

E 

0 

0 

1 

0 

1 
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17 i  tli  a  Stat e d  Pol  i  cy 


Proposal 

Number 

Ratinqs 

Orthogonal 

Vectors 

l 

A 

E 

E 

E 

A 

0 

1 

1 

1 

0 

2 

A 

E 

A 

E 

A 

0 

1 

0 

1 

0 

3 

A 

A 

E 

A 

A 

0 

0 

1 

0 

0 

4 

E 

E 

E 

A 

E 

1 

1 

1 

0 

1 

5 

E 

A 

E 

A 

E 

1 

0 

1 

0 

1 

6 

E 

E 

A 

A 

E 

1 

1 

0 

0 

1 

7 

A 

A 

A 

E 

A 

0 

0 

0 

1 

0 

8 

A 

A 

A 

A 

A 

0 

0 

0 

0 

0 

9 

A 

E 

E 

A 

A 

0 

1 

1 

0 

0 

10 

E 

A 

A 

E 

E 

1 

0 

0 

1 

0 

11 

A 

A 

E 

E 

A 

0 

0 

1 

1 

0 

12 

E 

E 

A 

E 

E 

1 

1 

0 

1 

1 

13 

A 

E 

A 

A 

A 

0 

1 

0 

0 

0 

14 

E 

A 

E 

E 

E 

1 

0 

1 

1 

1 

15 

E 

A 

A 

A 

E 

1 

0 

0 

0 

1 

16 

E 

E 

E 

E 

E 

1 

1 

1 

1 

1 

17 

E 

A 

A 

A 

A 

1 

0 

0 

0 

0 

18 

E 

A 

E 

A 

A 

1 

0 

1 

0 

0 

19 

E 

A 

E 

E 

A 

1 

0 

1 

1 

0 

20 

A 

E 

A 

E 

[- 

0 

1 

0 

1 

1 

21 

E 

E 

A 

r 

A 

1 

1 

0 

1 

0 

22 

E 

E 

E 

F 

1 

1 

1 

1 

0 

23 

A 

A 

E 

A 

E 

0 

0 

1 

0 

1 

24 

A 

E 

E 

E 

E 

0 

1 

1 

1 

1 

26 

A 

A 

A 

A 

E 

0 

0 

0 

0 

1 

26 

A 

A 

A 

E 

E 

0 

0 

0 

■ \ 

1 

1 

27 

E 

A 

A 

F 

A 

1 

0 

0 

1 

0 

28 

A 

A 

E 

E 

E 

0 

0 

1 

1 

1 

29 

A 

E 

A 

A 

F 

0 

1 

0 

0 

1 

30 

E 

E 

A 

A 

A 

1 

1 

0 

0 

0 

31 

F. 

E 

F. 

A 

A 

1 

1 

1 

0 

0 

32 

A 

E 

E 

A 

E 

0 

1 

1 

0 

1 
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SUBJECT.* 

TO: 


Source  Selection  Exercise 


20  Jun  3  979 


1 .  If  you  have  already  completed  and  returned  the 
source  selection  exercise,  please  disregard  this 
letter. 


2.  Due  to  TDY,  leave,  and  heavy  workloads,  th 
cutoff  date  for  completing  this  exercise  is  be 
extended.  Those  exercises  which  are  comoleted 
and  placed  in  the  mail  by  June  26  will  be 
included  in  flic  research,  i'lense  consider 
completing  the  exercise  by  this  date.  Additio 
exercises  will  be  greatly  appreciated.  Than): 
for  your  response. 


•'-eg 


sal 

you 


‘-'V 


f\ -  UiiC(y^ 

John  N.  Milligan,  Captain,  USAF 
APIT/ENS 
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APPENDIX  E 


CALCULATION  OF  F-TEST  VAI  UES 


87 


The  F-test  values  used  to  compare  regression  models  were  calculated 
using  the  following  formula: 


F 

o 


P 

Css  -  l  SS  .]/[(p-l)(k+l )] 
_ _  i=l  _ 

Cl  SS  . ]/[n-p(k+l )] 
j=l  J 


where 

SS  =  the  residual  sum  of  squares  derived  by  regressing 

e  all  compared  groups  of  decisions  together 

SSe .  =  the  residual  sum  of  squares  for  the  jth  group  of 

J  evaluations 

p  =  the  number  of  groups  being  compared 

k  =  the  number  of  cues 

n  =  the  total  number  of  decisions  in  all  groups  being 
compared 


The  null  hypothesis  is: 


Ho:  Bi  -  By  " 


B  ,  where  B .  =  B 

-p  -i  o 


B 


k 


The  alternate  hypothesis  is: 

H  :  B.  f  B.,  for  at  least  one  i,j  pair. 

a  — I  J 

The  null  hypothesis  is  rejected  if 

F0  >  Fa,  [ ( p-1 ) (k+1 )],  [n-p(k+l ) 
where  a  =  .05  for  all  comparisons  in  this  study. 


BB 


APPENDIX  F 


INDIVIDUAL  RELATIVE  WEIGHTS  AMD  R? 

In  this  Appendix  the  individual  relative  weights  and  Rz's  are 
listed.  On  the  case  numbers,  A  is  for  the  second  part  of  the  exercise 
(no  stated  policy  in  relation  to  the  evaluation  criteria),  while  B  is 
for  the  third  part  of  the  exercise  (with  a  stated  policy  in  relation  to 
the  evaluation  criteria). 


Relative  Weights  for  Cues 


Case 

Number 

Technical 

Operational 

Logistics 

Kan  aperient 

Cost 

_R?; _ 

1A 

.330 

.179 

.154 

.205 

.132 

.895 

IB 

.333 

.333 

.333 

.000 

.000 

1  .000 

2A 

.061 

.409 

.119 

.002 

.409 

.497 

2B 

.272 

.345 

.038 

.000 

.345 

.773 

3A 

.200 

.200 

.200 

.200 

.200 

.995 

3B 

.200 

.200 

.200 

.200 

.200 

.995 

4A 

.025 

.099 

.069 

.796 

.111 

.511 

4B 

.558 

.037 

.022 

.383 

.000 

.789 

5A 

.200 

.200 

.200 

.200 

.200 

1  .000 

5B 

.301 

.301 

.048 

.048 

.301 

.912 

6A 

.200 

.200 

.200 

.200 

.200 

1  .000 

6B 

.408 

.283 

.181 

.102 

.025 

.929 

7A 

.018 

.008 

.294 

.018 

.661 

.833 

7B 

.350 

.197 

.022 

.022 

.410 

.831 

8A 

.059 

.470 

.416 

.015 

.041 

.879 

8B 

.224 

.439 

.129 

.018 

.190 

.861 

9A 

.160 

.360 

.160 

.160 

.160 

.568 

9B 

.160 

.360 

.160 

.160 

.160 

.568 

10A 

.152 

.422 

.207 

.068 

.152 

.780 

10B 

.288 

.648 

.026 

.012 

.026 

.116 

11A 

.402 

.402 

.114 

.041 

.041 

.888 

11B 

.365 

.365 

.096 

.096 

.078 

.827 

12A 

.413 

.096 

.096 

.093 

.300 

.780 

12B 

.366 

.435 

.003 

.003 

.193 

.836 

13A 

.097 

.181 

.136 

.354 

.232 

.916 

13B 

.200 

.200 

.200 

.200 

.200 

1 .000 

14A 

.437 

.241 

.055 

.102 

.164 

.738 

14B 

.438 

.401 

.053 

.007 

.100 

.899 

15A 

.200 

.200 

.200 

.200 

.200 

1 .000 

15B 

.339 

.461 

.085 

.000 

.115 

.916 
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Case 

Number 

Technical 

Operational 

33A 

.251 

.251 

33  B 

.250 

.490 

34A 

.295 

.244 

34B 

.214 

.433 

35A 

.147 

.376 

35  B 

.302 

.428 

36A 

.099 

.099 

36  B 

.693 

.102 

37A 

.142 

.033 

37B 

.248 

.199 

38A 

.122 

.151 

33B 

.159 

.159 

39A 

.231 

.180 

39  B 

.187 

.494 

40A 

.093 

.093 

40B 

.321 

.570 

41 A 

.713 

.070 

41B 

.345 

.417 

42A 

.460 

.079 

42B 

.303 

.400 

43A 

.415 

.157 

43B 

.205 

.589 

44A 

.200 

.200 

44B 

.225 

.426 

45A 

.088 

.4  77 

45B 

.315 

.485 

46A 

.281 

.281 

46B 

.321 

.321 

47A 

.379 

.333 

47B 

.216 

.533 

48A 

.742 

.061 

48B 

.444 

.277 

49A 

.178 

.274 

49B 

.342 

.418 

Logistics 

Management 

Cost 

JlL 

.123 

.123 

.251 

.959 

.090 

.010 

.160 

.962 

.156 

.061 

.244 

.827 

.134 

.048 

.172 

.885 

.094 

.288 

.094 

.833 

.055 

.017 

.198 

.900 

.537 

.206 

.060 

.790 

.066 

.037 

.102 

.762 

.053 

.705 

.065 

.887 

.044 

.333 

.176 

.758 

.255 

.217 

.255 

.813 

.185 

.311 

.185 

.930 

.180 

.231 

.180 

.980 

.105 

.026 

.187 

.864 

.060 

.302 

.451 

.322 

.023 

.036 

.051 

.711 

.070 

.011 

.136 

.882 

.014 

.003 

.221 

.824 

.062 

.200 

.200 

.852 

.034 

.004 

.260 

.906 

.120 

.061 

.246 

.82? 

.031 

.003 

.171 

.882 

.200 

.200 

.200 

1  .000 

.173 

.088 

.088 

.780 

.119 

.119 

.197 

.873 

.013 

.041 

.146 

.921 

.281 

.079 

.079 

.900 

.018 

.018 

.321 

.923 

.120 

.095 

.073 

.983 

.083 

.049 

.110 

.965 

.000 

.136 

.061 

.280 

.060 

.011 

.208 

.674 

.240 

.034 

.274 

.831 

.024 

.004 

.213 

.838 
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VITA 


Captain  John  N.  Milligan  was  born  in  Oak  Ridge,  Tennessee,  on  January  4, 
1948.  He  graduated  from  Sandia  High  School  in  I960  at  Albuquerque,  New 
Mexico.  He  entered  the  University  of  New  Mexico  in  1956  and  graduated  in 
1971.  While  at  the  University,  he  majored  in  mathematics  and  minored  in 
chemistry.  He  was  a  member  of  Kappa  Mu  Epsilon,  mathematics  honorary,  and 
the  American  Chemical  Society.  He  was  also  a  member  and  officer  in  Phi  Ganna 
Delta,  a  national  social  fraternity.  In  1974,  Captain  Milligan  entered 
Louisiana  Tech  University  at  Bossier  City,  Louisiana.  In  August,  1979  he 
received  the  degree  of  Master  of  Business  Administration  from  Louisiana  Tech. 

Captain  Milligan  entered  the  United  States  Air  Force  in  June,  1971,  after 
being  commissioned  through  the  Reserve  Officer  Training  Corps  program.  His 
first  assignment  was  navigator  training  at  Mather  Air  Force  Base,  California. 
He  graduated  in  October,  1972.  Then,  he  entered  Navigator-Bombadier  Training 
at  Mather  Air  Force  Base.  After  graduation  in  February,  1973,  iie  was  assigned 
to  Barksdale  Air  Force  Base,  Louisiana.  He  has  completed  Squadron  Officer 
School  aid  Air  Command  and  Staff  College  through  correspondence  programs.  He 
was  assigned  to  the  Air  Force  Institute  of  Technology  in  August,  1978.  fol¬ 
lowing  graduation,  he  will  be  assigned  as  a.  Systems  Program  Manager  at  Wright- 
Patterson  Air  Force  Base,  Ohio. 

Permanent  Address:  7513  Gladden  Avenue,  Northeast 

Albuquerque  NM  87110 


